cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

lizou
by Contributor II
  • 2578 Views
  • 4 replies
  • 6 kudos

Resolved! Identity column definition lost using save as table

I found an issue:For a table with an identity column defined.when the table column is renamed using this method, the identity definition will be removed. That means using an identity column in a table requires extra attention to check whether the ide...

  • 2578 Views
  • 4 replies
  • 6 kudos
Latest Reply
lizou
Contributor II
  • 6 kudos

try to avoid reload table, I found we can upgrade table version, and use rename column commandALTER TABLE test_id2 SET TBLPROPERTIES (  'delta.columnMapping.mode' = 'name',  'delta.minReaderVersion' = '2',  'delta.minWriterVersion' = '6')ALTER TABLE ...

  • 6 kudos
3 More Replies
164079
by Contributor II
  • 2136 Views
  • 3 replies
  • 1 kudos

Resolved! unable to add new instance profile

Hi team, I want to start adding more instance profile per team . when adding it via TF , im getting the below error:Im able BTW to add and change other databricks resources via TF .This is my new code block:The new role created by the TF but wasnt ad...

image image image
  • 2136 Views
  • 3 replies
  • 1 kudos
Latest Reply
164079
Contributor II
  • 1 kudos

Thank you @Vivian Wilfred​ all ok now, with the databricks console and the TFHave a graet day!

  • 1 kudos
2 More Replies
User16752242161
by New Contributor II
  • 389 Views
  • 0 replies
  • 1 kudos

Hi,I am a Solutions Architect at Databricks Northern Europe. I had a deep dive demo session at Data + AI Work Tour in London on Nov 2, 2022. The title...

Hi,I am a Solutions Architect at Databricks Northern Europe. I had a deep dive demo session at Data + AI Work Tour in London on Nov 2, 2022. The title was How to Build Your Modern Data Stack on Databricks to Solve Modern Problems.In the above-mention...

  • 389 Views
  • 0 replies
  • 1 kudos
anandstarz
by New Contributor II
  • 1169 Views
  • 4 replies
  • 1 kudos

Certificate not received

I have passed Databricks Certified Data Engineer Associate on 29 October 2022 but still didn't receive the certificate, kindly help me on that.

  • 1169 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Anandhakumar R​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue.If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please visit the ...

  • 1 kudos
3 More Replies
Amol
by New Contributor II
  • 1120 Views
  • 4 replies
  • 0 kudos

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Dat...

I have passed my data analyst associate exam on 30th October , but still not received my certificate, its been more than than 4 days now , however Databricks mentioned in an email that will deliver within 24 hours. Can anyone help on this ?

  • 1120 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Amol Metkar​ Glad to hear that. Thanks for letting us know!

  • 0 kudos
3 More Replies
Anonymous
by Not applicable
  • 6637 Views
  • 9 replies
  • 7 kudos

Resolved! data frame takes unusually long time to write for small data sets

We have configured workspace with own vpc. We need to extract data from DB2 and write as delta format. we tried to for 550k records with 230 columns, it took 50mins to complete the task. 15mn records takes more than 18hrs. Not sure why this takes suc...

  • 6637 Views
  • 9 replies
  • 7 kudos
Latest Reply
elgeo
Valued Contributor II
  • 7 kudos

Hello. We face exactly the same issue. Reading is quick but writing takes long time. Just to clarify that it is about a table with only 700k rows. Any suggestions please? Thank youremote_table = spark.read.format ( "jdbc" ) \.option ( "driver" , "com...

  • 7 kudos
8 More Replies
RiyazAli
by Valued Contributor
  • 5054 Views
  • 3 replies
  • 7 kudos

Resolved! Converting a transformation written in Spark Scala to PySpark

Hello all,I've been tasked to convert a Scala Spark code to PySpark code with minimal changes (kinda literal translation).I've come across some code that claims to be a list comprehension. Look below for code snippet:%scala val desiredColumn = Seq("f...

  • 5054 Views
  • 3 replies
  • 7 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 7 kudos

Another follow-up question, if you don't mind. @Pat Sienkiewicz​ As I was trying to parse the name column into multiple columns. I came across the data below:("James,\"A,B\", Smith", "2018", "M", 3000)In order to parse these comma-included middle na...

  • 7 kudos
2 More Replies
Anonymous
by Not applicable
  • 1426 Views
  • 4 replies
  • 19 kudos

Resolved! How to prepare for Databricks Data Engineer Professional

Hi all,Could you please help suggest me some resource to prepare for " Databricks Data Engineer Professional" exam?I have also take the course in Databricks Accademy but seems not enough for this exam?Thank you so much!!!Best Regards,Nhan Nguyen

  • 1426 Views
  • 4 replies
  • 19 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 19 kudos

waitting some one post more details and experience road map to take exam

  • 19 kudos
3 More Replies
Rajesh2747
by New Contributor III
  • 1975 Views
  • 3 replies
  • 4 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 1975 Views
  • 3 replies
  • 4 kudos
Latest Reply
Rajesh2747
New Contributor III
  • 4 kudos

@Hubert Dudek​ Thanks for the information. It is working.I have one more question:Is notebook saves changes automatically ?? If yes, I can see 'save now' option on top right corner(where user details shows/Version details shows) what is use of that o...

  • 4 kudos
2 More Replies
Sandy21
by New Contributor III
  • 809 Views
  • 2 replies
  • 3 kudos

Queries with running REST API command in databricks to create a Job

What happens when jobs/create REST API command is run multiple times(say 3 times) with the same JSON configuration? Will 3 jobs are created with the same name or only 1 job will be created?

  • 809 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Santhosh Raj​ , We haven’t heard from you since the last response from @Debayan Mukherjee​ ​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
1 More Replies
Rajesh2747
by New Contributor III
  • 4764 Views
  • 2 replies
  • 3 kudos

Resolved! How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

How to check, who did changes in databrics notebook and what are changes done in databrics notebook.

  • 4764 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Rajesh Gowd​ , We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Also, P...

  • 3 kudos
1 More Replies
ae20cg
by New Contributor III
  • 4356 Views
  • 4 replies
  • 3 kudos

Databricks web terminal not able to parse notebooks.

In the web terminal, I am not able to search for text for example using grep -l "search_term" db_notebookThis leads to "Operation not permitted on <notebook> error", any ideas why?This is for all DB notebooks in my cluster.Thanks!

  • 4356 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Andrej Erkelens​ ​, We haven’t heard from you since the last response from @Debayan Mukherjee​​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be help...

  • 3 kudos
3 More Replies
rams
by Contributor
  • 2084 Views
  • 4 replies
  • 4 kudos

Rollback error - Configuring Databricks lakehouse platform with AWS account

I have logged in databricks account and while creating the workspace i have chosen quickstart approach to configure the databricks with AWS. During the quickstart process the databricks page will redirect to aws cloudformation stack page where the ac...

  • 2084 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @rams shonu​ ​, We haven’t heard from you since the last response from @Pat Sienkiewicz​​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 4 kudos
3 More Replies
Serhii
by Contributor
  • 1450 Views
  • 2 replies
  • 0 kudos

Resolved! Horovod Databricks Job - custom module not found error

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...

  • 1450 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Sergii Ivakhno​ ​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Als...

  • 0 kudos
1 More Replies
Sufyan
by New Contributor II
  • 985 Views
  • 3 replies
  • 2 kudos

Resolved! When and where would I receive my Certificate after passing the exam?

I have passed the Databricks Certified Data Analyst Associate exam on 29 October 2022. I have not received the corticate yet.

  • 985 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Sufyan Shafique​ We are really sorry for the delays.Just a friendly- follow-up, have you got your certification and badge? If yes, please mark the answer as best.Thanks and Regards

  • 2 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels