cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Akshith_Rajesh
by • New Contributor III
  • 2204 Views
  • 4 replies
  • 1 kudos

Does DataBricks lock the file in Adls Gen 2 before writing (Append) to a file If yes then how can we fetch the file is locked

I have a requirement , I am running 2 Notebooks parallelly I want to overwrite the file parallelly .If 2 Notebooks Try to overwrite the file at the same time , will I lose the data because of overwriting the file at the same time .I want to overwr...

Data Engineering
Reply
  • 2204 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Rajesh Akshith​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
elgeo
by • Valued Contributor II
  • 1260 Views
  • 1 replies
  • 2 kudos

Moving data to a delta table keeping the old surrogate ids intact

Hello experts! We have a table in our current system that we need to move it (one-off) to a delta in Databricks keeping its Ids (surrogate keys) intact. We think to of the following steps: 1. create a new delta table with a "BIGINT GENERATED BY DEFAU...

Data Engineering
Reply
  • 1260 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
lizou
Contributor II
  • 2 kudos

same here, I submitted an idea in the azure databricks portalhttps://feedback.azure.com/d365community/idea/d403303c-6761-ed11-a81b-000d3ae5ae95SET IDENTITY_INSERT ONwhen a column is defined as GENERATED ALWAYS, we often need to reload data with exact...

  • 2 kudos
577391
by • New Contributor II
  • 2102 Views
  • 2 replies
  • 0 kudos

Resolved! How do I merge two tables and track changes to missing rows as well as new rows

In my scenario, the new data coming in are the current, valid records. Any records that are not in the new data should be labeled as 'Gone", any matching records should be labeled with "Updated". And finally, any new records should be added.So in sum...

Data Engineering
Reply
  • 2102 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Detection deletions does not work out of the box.The merge statement will evaluate the incoming data against the existing data. It will not check the existing data against the incoming data.To mark deletions, you will have to specifically update tho...

  • 0 kudos
1 More Replies
Labels