cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

anandstarz
by New Contributor II
  • 1176 Views
  • 4 replies
  • 1 kudos

Certificate not received

I have passed Databricks Certified Data Engineer Associate on 29 October 2022 but still didn't receive the certificate, kindly help me on that.

  • 1176 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Anandhakumar R​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue.If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please visit the ...

  • 1 kudos
3 More Replies
Amol
by New Contributor II
  • 1135 Views
  • 4 replies
  • 0 kudos

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Dat...

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Databricks mentioned in an email that will deliver within 24 hours. Can anyone help on this ?

  • 1135 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Amol Metkar​ Glad to hear that. Thanks for letting us know!

  • 0 kudos
3 More Replies
Anonymous
by Not applicable
  • 6697 Views
  • 9 replies
  • 7 kudos

Resolved! data frame takes unusually long time to write for small data sets

We have configured workspace with own vpc. We need to extract data from DB2 and write as delta format. we tried to for 550k records with 230 columns, it took 50mins to complete the task. 15mn records takes more than 18hrs. Not sure why this takes suc...

  • 6697 Views
  • 9 replies
  • 7 kudos
Latest Reply
elgeo
Valued Contributor II
  • 7 kudos

Hello. We face exactly the same issue. Reading is quick but writing takes long time. Just to clarify that it is about a table with only 700k rows. Any suggestions please? Thank youremote_table = spark.read.format ( "jdbc" ) \.option ( "driver" , "com...

  • 7 kudos
8 More Replies
RiyazAli
by Valued Contributor
  • 5085 Views
  • 3 replies
  • 7 kudos

Resolved! Converting a transformation written in Spark Scala to PySpark

Hello all,I've been tasked to convert a Scala Spark code to PySpark code with minimal changes (kinda literal translation).I've come across some code that claims to be a list comprehension. Look below for code snippet:%scala val desiredColumn = Seq("f...

  • 5085 Views
  • 3 replies
  • 7 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 7 kudos

Another follow-up question, if you don't mind. @Pat Sienkiewicz​ As I was trying to parse the name column into multiple columns. I came across the data below:("James,\"A,B\", Smith", "2018", "M", 3000)In order to parse these comma-included middle na...

  • 7 kudos
2 More Replies
Anonymous
by Not applicable
  • 1435 Views
  • 4 replies
  • 19 kudos

Resolved! How to prepare for Databricks Data Engineer Professional

Hi all,Could you please help suggest me some resource to prepare for " Databricks Data Engineer Professional" exam?I have also take the course in Databricks Accademy but seems not enough for this exam?Thank you so much!!!Best Regards,Nhan Nguyen

  • 1435 Views
  • 4 replies
  • 19 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 19 kudos

waitting some one post more details and experience road map to take exam

  • 19 kudos
3 More Replies
Rajesh2747
by New Contributor III
  • 2001 Views
  • 3 replies
  • 4 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 2001 Views
  • 3 replies
  • 4 kudos
Latest Reply
Rajesh2747
New Contributor III
  • 4 kudos

@Hubert Dudek​ Thanks for the information. It is working.I have one more question:Is notebook saves changes automatically ?? If yes, I can see 'save now' option on top right corner(where user details shows/Version details shows) what is use of that o...

  • 4 kudos
2 More Replies
Sandy21
by New Contributor III
  • 818 Views
  • 2 replies
  • 3 kudos

Queries with running REST API command in databricks to create a Job

What happens when jobs/create REST API command is run multiple times(say 3 times) with the same JSON configuration? Will 3 jobs are created with the same name or only 1 job will be created?

  • 818 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Santhosh Raj​ , We haven’t heard from you since the last response from @Debayan Mukherjee​ ​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
1 More Replies
Rajesh2747
by New Contributor III
  • 4824 Views
  • 2 replies
  • 3 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 4824 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Rajesh Gowd​ , We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Also, P...

  • 3 kudos
1 More Replies
ae20cg
by New Contributor III
  • 4379 Views
  • 4 replies
  • 3 kudos

Databricks web terminal not able to parse notebooks.

In the web terminal, I am not able to search for text for example using grep -l "search_term" db_notebookThis leads to "Operation not permitted on <notebook> error", any ideas why?This is for all DB notebooks in my cluster.Thanks!

  • 4379 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Andrej Erkelens​ ​, We haven’t heard from you since the last response from @Debayan Mukherjee​​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
3 More Replies
rams
by Contributor
  • 2108 Views
  • 4 replies
  • 4 kudos

Rollback error - Configuring Databricks lakehouse platform with AWS account

I have logged in databricks account and while creating the workspace i have chosen quickstart approach to configure the databricks with AWS. During the quickstart process the databricks page will redirect to aws cloudformation stack page where the ac...

  • 2108 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @rams shonu​ ​, We haven’t heard from you since the last response from @Pat Sienkiewicz​​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 4 kudos
3 More Replies
Serhii
by Contributor
  • 1467 Views
  • 2 replies
  • 0 kudos

Resolved! Horovod Databricks Job - custom module not found error

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...

  • 1467 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Sergii Ivakhno​ ​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Als...

  • 0 kudos
1 More Replies
Sufyan
by New Contributor II
  • 999 Views
  • 3 replies
  • 2 kudos

Resolved! When and where would I receive my Certificate after passing the exam?

I have passed the Databricks Certified Data Analyst Associate exam on 29 October 2022. I have not received the corticate yet.

  • 999 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Sufyan Shafique​ We are really sorry for the delays.Just a friendly- follow-up, have you got your certification and badge? If yes, please mark the answer as best.Thanks and Regards

  • 2 kudos
2 More Replies
jefft
by New Contributor III
  • 1801 Views
  • 3 replies
  • 1 kudos

Databricks integration with DocumentDB

Hi Everyone,I was wondering if anyone here has any experience or tips reading data from AWS DocumentDB. I am working on this using the MongoDB connector. For DocumentDB we also need to work with the required creds issued as a .pem file by AWS. Th...

  • 1801 Views
  • 3 replies
  • 1 kudos
Latest Reply
jefft
New Contributor III
  • 1 kudos

Hi @Kaniz Fatma​ ,Thank you so much for your response. Your suggestions were helpful. As per the AWS documentation, DocumentDB is MongoDB compatible. "With Amazon DocumentDB, you can run the same application code and use the same drivers and tools th...

  • 1 kudos
2 More Replies
StanleyTang
by New Contributor III
  • 1675 Views
  • 4 replies
  • 4 kudos

How to run SQL queries from services when data migrated from SQL server to data lake?

Currently our service provides an API to serve the purchase records. The purchase records are stored in SQL database. To simplify, when users want to get their recent purchase records, they make an API call. The API call will run a SQL query on the D...

  • 1675 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Stanley Tang​​​, We haven’t heard from you since the last response from @Debayan Mukherjee​ , and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful...

  • 4 kudos
3 More Replies
Sanjujain
by New Contributor II
  • 915 Views
  • 2 replies
  • 1 kudos

Resolved! I have created account but forgot to check community edition option while signing up. How this can be solved now.

I have created account but forgot to check community edition option while signing up. How this can be solved now.

  • 915 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @serg-v (Customer)​​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.A...

  • 1 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels