cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alesventus
by • Contributor
  • 1811 Views
  • 0 replies
  • 0 kudos

Performance issue: Running 50 notebooks from ADF

I have process in Data factory, that loads CDC changes from sql server and then trigger notebook with merge to bronze and silver zone. Single notebook takes about 1 minute to run but when all 50 notebooks are fired at once the whole process takes 25 ...

Data Engineering
performance issue
Reply
  • 1811 Views
  • 0 replies
  • 0 kudos
Greg
by • New Contributor III
  • 2392 Views
  • 1 replies
  • 4 kudos

How to reduce storage space consumed by delta with many updates

I have 1 delta table that I continuously append events into, and a 2nd delta table that I continuously merge into (streamed from the 1st table) that has unique ID's where properties are updated from the events (An ID represents a unique thing that ge...

Data Engineering
Reply
  • 2392 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Jb11
New Contributor II
  • 4 kudos

Did you already solved this problem?

  • 4 kudos
bfridley
by • New Contributor II
  • 4363 Views
  • 2 replies
  • 0 kudos

DLT Pipeline Out Of Memory Errors

I have a DLT pipeline that has been running for weeks. Now, trying to rerun the pipeline with the same code and same data fails. I've even tried updating the compute on the cluster to about 3x of what was previously working and it still fails with ou...

bfridley_1-1695328329708.png bfridley_2-1695328372419.png
Data Engineering
Reply
  • 4363 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
rajib_bahar_ptg
New Contributor III
  • 0 kudos

I'd focus on understanding the codebase first. It'll help you decide what logic or data asset to keep or not keep when you try to optimize it. If you share the architecture of the application, the problem it solves, and some sample code here, it'll h...

  • 0 kudos
1 More Replies
gkrilis
by • New Contributor
  • 8107 Views
  • 1 replies
  • 0 kudos

How to stop SparkSession within notebook without errr

I want to run an ETL job and when the job ends I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling spark.stop() the job returns with status failed even though it has f...

Data Engineering
cluster
SparkSession
Reply
  • 8107 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
PremadasV
New Contributor II
  • 0 kudos

Please refer to this Job fails, but Apache Spark tasks finish - Databricks

  • 0 kudos
Gilg
by • Contributor II
  • 1344 Views
  • 0 replies
  • 0 kudos

Add data manually to DLT

Hi Team,Is there a way that we can add data manually to the tables that are generated by DLT?We have done a PoC using DLT for Sep 15 to current data. Now, that they are happy, they wanted the previous data from Synapse and put into Databricks.I can e...

Data Engineering
Reply
  • 1344 Views
  • 0 replies
  • 0 kudos
Martin1
by • New Contributor II
  • 11019 Views
  • 2 replies
  • 1 kudos

Referring to Azure Keyvault secrets in spark config

Hi allIn spark config for a cluster, it works well to refer to a Azure Keyvault secret in the "value" part of the name/value combo on a config row/setting.For example, this works fine (I've removed the string that is our specific storage account name...

Data Engineering
Reply
  • 11019 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
kp12
New Contributor II
  • 1 kudos

Hello,Is there any update on this issue please? Databricks no longer recommend mounting external location, so the other way to access Azure storage is to use spark config as mentioned in this document - https://learn.microsoft.com/en-us/azure/databri...

  • 1 kudos
1 More Replies
scrimpton
by • New Contributor II
  • 4418 Views
  • 0 replies
  • 0 kudos

Permanently delete dropped table (Unity Catalog)

The recommendation before dropping a table is to do a DELETE then VACUUM RETENTION 0 (recommended in DEV).If you DROP the table without doing a DELETE|VACUUM, your table will be soft deleted with your entire data (permanently deletedin 30 days) and y...

Data Engineering
Reply
  • 4418 Views
  • 0 replies
  • 0 kudos
scrimpton
by • New Contributor II
  • 2730 Views
  • 0 replies
  • 0 kudos

Statistics for rearranged columns

The table property dataSkippingNumIndexedCols that gets statistics for a table starts from left to right. I am wondering what will happen to the statistics for both new and old records if we add a column in between using FIRST|AFTER identifier. 

Vince_03_0-1696039970438.png
Data Engineering
Reply
  • 2730 Views
  • 0 replies
  • 0 kudos
marvin1
by • New Contributor III
  • 631 Views
  • 0 replies
  • 0 kudos

Bamboolib error

What is the status of bamboolib?  I understand that it is public preview but I'm unable to find any support references.  I am getting error below.  I've tried installing in a notebook, on a cluster, creating a pandas dataframe and running bam, etc.  ...

Data Engineering
Reply
  • 631 Views
  • 0 replies
  • 0 kudos
938452
by • New Contributor III
  • 8667 Views
  • 0 replies
  • 0 kudos

Spark is not reading Kinesis Data as fast as specified

Hi Databricks community team,I have code as below"""df = spark.readStream \.format("kinesis") \.option("endpointUrl", endpoint_url) \.option("streamName", stream_name) \.option("initialPosition", "latest") \.option("consumerMode", "efo") \.option("ma...

Data Engineering
Reply
  • 8667 Views
  • 0 replies
  • 0 kudos
mbvb_py
by • New Contributor II
  • 5078 Views
  • 4 replies
  • 0 kudos

Create cluster error: Backend service unavailable

hello,i'm new to Databricks (community edition account) and encountered a problem just now.When creating a new cluster (default 10.4 LTS) it fails with the following error: Backend service unavailable.I've tried a different runtime > same issue.I've ...

Data Engineering
Reply
  • 5078 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
stefnhuy
New Contributor III
  • 0 kudos

Hey mbvb_py,I'm sorry to hear you're facing this "Backend service unavailable" issue with Databricks. I've encountered similar problems in the past, and it can be frustrating. Don't worry; you're not alone in this!From my experience, this error can o...

  • 0 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels