cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Amol
by New Contributor II
  • 1135 Views
  • 4 replies
  • 0 kudos

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Dat...

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Databricks mentioned in an email that will deliver within 24 hours. Can anyone help on this ?

  • 1135 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Amol Metkar​ Glad to hear that. Thanks for letting us know!

  • 0 kudos
3 More Replies
Anonymous
by Not applicable
  • 6718 Views
  • 9 replies
  • 7 kudos

Resolved! data frame takes unusually long time to write for small data sets

We have configured workspace with own vpc. We need to extract data from DB2 and write as delta format. we tried to for 550k records with 230 columns, it took 50mins to complete the task. 15mn records takes more than 18hrs. Not sure why this takes suc...

  • 6718 Views
  • 9 replies
  • 7 kudos
Latest Reply
elgeo
Valued Contributor II
  • 7 kudos

Hello. We face exactly the same issue. Reading is quick but writing takes long time. Just to clarify that it is about a table with only 700k rows. Any suggestions please? Thank youremote_table = spark.read.format ( "jdbc" ) \.option ( "driver" , "com...

  • 7 kudos
8 More Replies
RiyazAli
by Valued Contributor
  • 5098 Views
  • 3 replies
  • 7 kudos

Resolved! Converting a transformation written in Spark Scala to PySpark

Hello all,I've been tasked to convert a Scala Spark code to PySpark code with minimal changes (kinda literal translation).I've come across some code that claims to be a list comprehension. Look below for code snippet:%scala val desiredColumn = Seq("f...

  • 5098 Views
  • 3 replies
  • 7 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 7 kudos

Another follow-up question, if you don't mind. @Pat Sienkiewicz​ As I was trying to parse the name column into multiple columns. I came across the data below:("James,\"A,B\", Smith", "2018", "M", 3000)In order to parse these comma-included middle na...

  • 7 kudos
2 More Replies
Anonymous
by Not applicable
  • 1444 Views
  • 4 replies
  • 19 kudos

Resolved! How to prepare for Databricks Data Engineer Professional

Hi all,Could you please help suggest me some resource to prepare for " Databricks Data Engineer Professional" exam?I have also take the course in Databricks Accademy but seems not enough for this exam?Thank you so much!!!Best Regards,Nhan Nguyen

  • 1444 Views
  • 4 replies
  • 19 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 19 kudos

waitting some one post more details and experience road map to take exam

  • 19 kudos
3 More Replies
Rajesh2747
by New Contributor III
  • 2012 Views
  • 3 replies
  • 4 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 2012 Views
  • 3 replies
  • 4 kudos
Latest Reply
Rajesh2747
New Contributor III
  • 4 kudos

@Hubert Dudek​ Thanks for the information. It is working.I have one more question:Is notebook saves changes automatically ?? If yes, I can see 'save now' option on top right corner(where user details shows/Version details shows) what is use of that o...

  • 4 kudos
2 More Replies
Sandy21
by New Contributor III
  • 819 Views
  • 2 replies
  • 3 kudos

Queries with running REST API command in databricks to create a Job

What happens when jobs/create REST API command is run multiple times(say 3 times) with the same JSON configuration? Will 3 jobs are created with the same name or only 1 job will be created?

  • 819 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Santhosh Raj​ , We haven’t heard from you since the last response from @Debayan Mukherjee​ ​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
1 More Replies
Rajesh2747
by New Contributor III
  • 4853 Views
  • 2 replies
  • 3 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 4853 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Rajesh Gowd​ , We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Also, P...

  • 3 kudos
1 More Replies
ae20cg
by New Contributor III
  • 4382 Views
  • 4 replies
  • 3 kudos

Databricks web terminal not able to parse notebooks.

In the web terminal, I am not able to search for text for example using grep -l "search_term" db_notebookThis leads to "Operation not permitted on <notebook> error", any ideas why?This is for all DB notebooks in my cluster.Thanks!

  • 4382 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Andrej Erkelens​ ​, We haven’t heard from you since the last response from @Debayan Mukherjee​​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
3 More Replies
rams
by Contributor
  • 2112 Views
  • 4 replies
  • 4 kudos

Rollback error - Configuring Databricks lakehouse platform with AWS account

I have logged in databricks account and while creating the workspace i have chosen quickstart approach to configure the databricks with AWS. During the quickstart process the databricks page will redirect to aws cloudformation stack page where the ac...

  • 2112 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @rams shonu​ ​, We haven’t heard from you since the last response from @Pat Sienkiewicz​​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 4 kudos
3 More Replies
Serhii
by Contributor
  • 1469 Views
  • 2 replies
  • 0 kudos

Resolved! Horovod Databricks Job - custom module not found error

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...

  • 1469 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Sergii Ivakhno​ ​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Als...

  • 0 kudos
1 More Replies
Sufyan
by New Contributor II
  • 1002 Views
  • 3 replies
  • 2 kudos

Resolved! When and where would I receive my Certificate after passing the exam?

I have passed the Databricks Certified Data Analyst Associate exam on 29 October 2022. I have not received the corticate yet.

  • 1002 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Sufyan Shafique​ We are really sorry for the delays.Just a friendly- follow-up, have you got your certification and badge? If yes, please mark the answer as best.Thanks and Regards

  • 2 kudos
2 More Replies
jefft
by New Contributor III
  • 1805 Views
  • 3 replies
  • 1 kudos

Databricks integration with DocumentDB

Hi Everyone,I was wondering if anyone here has any experience or tips reading data from AWS DocumentDB. I am working on this using the MongoDB connector. For DocumentDB we also need to work with the required creds issued as a .pem file by AWS. Th...

  • 1805 Views
  • 3 replies
  • 1 kudos
Latest Reply
jefft
New Contributor III
  • 1 kudos

Hi @Kaniz Fatma​ ,Thank you so much for your response. Your suggestions were helpful. As per the AWS documentation, DocumentDB is MongoDB compatible. "With Amazon DocumentDB, you can run the same application code and use the same drivers and tools th...

  • 1 kudos
2 More Replies
StanleyTang
by New Contributor III
  • 1681 Views
  • 4 replies
  • 4 kudos

How to run SQL queries from services when data migrated from SQL server to data lake?

Currently our service provides an API to serve the purchase records. The purchase records are stored in SQL database. To simplify, when users want to get their recent purchase records, they make an API call. The API call will run a SQL query on the D...

  • 1681 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Stanley Tang​​​, We haven’t heard from you since the last response from @Debayan Mukherjee​ , and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful...

  • 4 kudos
3 More Replies
Sanjujain
by New Contributor II
  • 916 Views
  • 2 replies
  • 1 kudos

Resolved! I have created account but forgot to check community edition option while signing up. How this can be solved now.

I have created account but forgot to check community edition option while signing up. How this can be solved now.

  • 916 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @serg-v (Customer)​​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.A...

  • 1 kudos
1 More Replies
Sajid1
by Contributor
  • 4122 Views
  • 6 replies
  • 1 kudos
  • 4122 Views
  • 6 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Sajid Thavalengal Rahiman​, We haven’t heard from you since the last response from @Laurent Léturgez​ ​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it c...

  • 1 kudos
5 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels