cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Deba
by New Contributor II
  • 426 Views
  • 0 replies
  • 1 kudos

2023 DataAISummit

Started with the keynote session and instantly got two ideas Safe by design and data lineage by design, excited with changes for spark3.4 and lakehouseIQ

  • 426 Views
  • 0 replies
  • 1 kudos
RamKar
by New Contributor
  • 350 Views
  • 0 replies
  • 0 kudos

Unity Catalog

Looking forward to the ease of data security and sharing 

  • 350 Views
  • 0 replies
  • 0 kudos
DataengineerAMA
by New Contributor
  • 1523 Views
  • 2 replies
  • 0 kudos

UnityCatalog

Can we get the lineage for onprem spark application running i. Loval

  • 1523 Views
  • 2 replies
  • 0 kudos
Latest Reply
Deeprpatel2
New Contributor II
  • 0 kudos

Yes, you can create private endpoint links and application access (depends on your cloud provider) to connect with your on-premise systems and unity catalog can sync that well. Lineage is supported for all languages and is captured down to the column...

  • 0 kudos
1 More Replies
Peston
by New Contributor
  • 2191 Views
  • 0 replies
  • 0 kudos

DataAI Summit

Having a great time at the summit and learning about the advances with AI. The advantages of adding the ability to use english as a new programming language and the advantage that it will bring to the companies that adapt to the future of Databricks....

  • 2191 Views
  • 0 replies
  • 0 kudos
vrakesh77
by New Contributor
  • 524 Views
  • 0 replies
  • 0 kudos

DAIS 2023

Amazing! Excited to be part of my first DAIS. Incredible contributions. Looking forward to putting some of the tools and features into practice.

  • 524 Views
  • 0 replies
  • 0 kudos
Ncman
by New Contributor
  • 529 Views
  • 0 replies
  • 0 kudos

Summit

Having a great time at the data summit! It is an amazing experience and is organized very well! #databricks

  • 529 Views
  • 0 replies
  • 0 kudos
bikash84
by New Contributor III
  • 3495 Views
  • 1 replies
  • 1 kudos

Resolved! need help with Concurrent append exception during delta merge

I have 2 datasets getting loaded into a common silver table. These are event driven and notebooks are triggered when a file is dropped into the storage account. When the files come in at the same time, one dataset fails with concurrent append excepti...

  • 3495 Views
  • 1 replies
  • 1 kudos
Latest Reply
yuvapraveen_k
New Contributor III
  • 1 kudos

Databricks provides ACID guarantees since the inception of Delta format. In order to ensure the C - Consistency is addressed, it limits concurrent workflows to perform updates at the same time, like other ACID compliant SQL engines. The key differenc...

  • 1 kudos
Nam
by New Contributor II
  • 2424 Views
  • 1 replies
  • 0 kudos
  • 2424 Views
  • 1 replies
  • 0 kudos
Latest Reply
AlanF
New Contributor II
  • 0 kudos

No difference, both are very similar only difference is that in the SQL editor you cannot take advantage of things like SQL serverless clusters, better visibility of data objects and a more simplified exploration tooling. SQL in notebooks mostly run ...

  • 0 kudos
Essjay
by New Contributor II
  • 2432 Views
  • 1 replies
  • 3 kudos

Resolved! Standard SKU to Premium SKU?

What's the effort involved with converting a databricks standard workspace to a premium? Is it 1 click of a button? Or are there other considerations you need to think about?

  • 2432 Views
  • 1 replies
  • 3 kudos
Latest Reply
srusso
New Contributor III
  • 3 kudos

It’s pretty easy! You can use the Azure API or the CLI. To upgrade, use the Azure Databricks workspace creation API to recreate the workspace with exactly the same parameters as the Standard workspace, specifying the sku property as Premium. To use t...

  • 3 kudos
msa_2j212
by New Contributor II
  • 2631 Views
  • 2 replies
  • 1 kudos

Downloading and storing a PDF file to FileStore not working

I'm trying to download a PDF file and store it in FileStore using this code in a Notebook: with open('/dbfs/FileStore/file.pdf', 'wb') as f: f.write(requests.get('https://url.com/file.pdf').content) But I'm getting this error:FileNotFoundError: [...

  • 2631 Views
  • 2 replies
  • 1 kudos
Latest Reply
msa_2j212
New Contributor II
  • 1 kudos

This worked, thanks. 

  • 1 kudos
1 More Replies
berdoni
by New Contributor III
  • 937 Views
  • 0 replies
  • 0 kudos

Migrate an Azure Delta Table from Standard to Premium Storage

Hello,I'm exercising a migration of an azure delta table (10TB) from Azure Standard  performance tier to Azure Premium. The plan is to create a new storage account and copy the table into it. Then we will switch to the new table. The table contains r...

Data Engineering
azure
deep clone
  • 937 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels