- 1623 Views
- 0 replies
- 0 kudos
BEGINNER
Hey i am beginner and i am intrested to learn data engineering path for azure databrics.Can some one please provide some guidance
- 1623 Views
- 0 replies
- 0 kudos
Forum Posts
- 2781 Views
- 1 replies
- 2 kudos
How to get the normalized hours value for the Databricks EMR cluster?
Hi Hubert!I'm working on a usecase to get the compute usage stats. I used boto3 code and describe_cluster() function to get the normalized instance hours value for the EMR cluster. I would like to know if there is an equivalent for this normalized in...
- 2781 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Bala, nice to meet you.Yes, there is an equivalent. For example, in Databricks, you need to use the get method in cluster API or CLI implementation of API.https://docs.databricks.com/dev-tools/api/latest/clusters.html#get
- 2 kudos
- 28402 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 28402 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos
- 4000 Views
- 2 replies
- 2 kudos
Simba ODBC datetime with millisecond overflows
Hello,Using odbc 2.6.24.1041-2 for Linux, when inserting rows with milliseconds precision date (e.g. 2022-07-03 13:57:48.500) precision I get:2022/07/03 14:41:19 SQLExecute: {22008} [Simba][Support] (40520) Datetime field overflow resulting from inva...
- 4000 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
I was passing a string (e.g. '2022-07-03 13:57:48.500') to the Golang SQL driver which is not working if the ms part is specified, but otherwise it works (e.g. '2022-07-03 13:57:48'). Passing a native Golang time.Time seems to work for timestamps wit...
- 2 kudos
- 1490 Views
- 0 replies
- 5 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...
- 1490 Views
- 0 replies
- 5 kudos
- 2212 Views
- 0 replies
- 0 kudos
Unexpected behaviour when creating multiple Account-Level Accounts
We've had a qustion regarding possibly unexpected behaviour when creating multiple accounts on the account-level on https://accounts.cloud.databricks.com/.Short Version:It's possible to create multiple accounts with different letter-cases on https://...
- 2212 Views
- 0 replies
- 0 kudos
- 6530 Views
- 3 replies
- 1 kudos
Resolved! ipywidgets not working in DBR 11.0 on Community Edition
I'm looking forward to using ipywidgets which should be working in DBR 11.0 as they provide more options when creating a notebook UI. I saw that DBR 11.0 is available as of yesterday so I created a test cluster in the Databricks Community Edition ju...
- 6530 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Wayne Deleersnyder​ I was able to import ipywidgets in DBR 11.0. As you can see in the output below. The slider is visibleYou are facing this issue probably because the community edition has limited access. To get all the features you should at l...
- 1 kudos
- 2395 Views
- 3 replies
- 2 kudos
pd.read_csv failed
https://i.imgur.com/PAGzSr9.png
- 2395 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Thanks for your kind reply:Below works for me:https://imgur.com/BmMzatIBut why, as you mentioned, using the classic path, below does not work?https://imgur.com/Ba1a4Iv
- 2 kudos
- 1821 Views
- 0 replies
- 0 kudos
How to make a paper airplane
How to make a paper airplane
- 1821 Views
- 0 replies
- 0 kudos
- 2851 Views
- 2 replies
- 0 kudos
DBrick workspace URL block outside the corporate network?
As per security concern, need to restrict/block the dbricks workspace url outside the corporate network. Tried below ip access list, it able to restrict only user login access out the corporate network but still the workspace id url is live outside t...
- 2851 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @as999​ Hope everything is going great!Does @Atanu Sarkar​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. We'...
- 0 kudos
- 6619 Views
- 3 replies
- 1 kudos
How do you install R packages?
I've tried this, but it doesn't appear to be working: https://community.databricks.com/s/question/0D53f00001GHVX1CAP/unable-to-install-sf-and-rgeos-r-packages-on-the-clusterWhen I run the following after that init script, I receive an error.library(r...
- 6619 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hey there @Christopher Flach​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 1 kudos
- 8832 Views
- 5 replies
- 0 kudos
Resolved! A void column was created after connecting to cosmos
Hi everyone, I have connected to Cosmos using this tutorial https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3_2-12/Samples/DatabricksLiveContainerMigrationAfter creating a table using a simple SQL command:CREATE TA...
- 8832 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey there @Joel iemma​ Hope all is well! Just wanted to check in if you would be happy to mark an answer as best for us, please? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 5398 Views
- 2 replies
- 0 kudos
Resolved! JDBC Connectivity via workspace url when No Public IP selected.
Hi All, I think I might be missing something in regard to No Pubic IP Clusters. I have set this option on a workspace (Azure) and setup the appropriate subnets. To my surprise, when I went to setup a JDBC connection to the cluster the JDBC connec...
- 5398 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey there @Ashley Betts​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 27918 Views
- 18 replies
- 13 kudos
Resolved! HELP! Converting GZ JSON to Delta causes massive CPU spikes and ETL's take days!
Hi there,I was wondering if I could get your advise.We would like to create a bronze delta table using GZ JSON data stored in S3 but each time we attempt to read and write it our clusters CPU spikes to 100%. We are not doing any transformations but s...
- 27918 Views
- 18 replies
- 13 kudos
Latest Reply
- 13 kudos
Hi Kaniz,Thanks for the note and thank you everyone for the suggestions and help. @Joseph Kambourakis​ I aded your suggestion to our load but I did not see any change in how our data loads or the time it takes to load data. I've done some additional ...
- 13 kudos
- 14218 Views
- 3 replies
- 4 kudos
Resolved! Recommended way to integrate MongoDB as a streaming source
Current state:Data is stored in MongoDB Atlas which is used extensively by all servicesData lake is hosted in same AWS region and connected to MongoDB over private link Requirements:Streaming pipelines that continuously ingest, transform/analyze and ...
- 14218 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Another option if you'd like to use Spark as the ingestion is to use the new Spark Connector V10.0 which support Spark Structured Streaming. https://www.mongodb.com/developer/languages/python/streaming-data-apache-spark-mongodb/. If you use Kafka, th...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
