cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SaraCorralLou
by • New Contributor III
  • 2598 Views
  • 1 replies
  • 0 kudos

Clean driver during notebook execution

Is there any way to clear the memory driver during the execution of my notebook? I have several functions that are executed in the driver and that generate in it different dataframes that are not necessary (these dataframes are created just to do som...

Data Engineering
Reply
  • 2598 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

since spark uses lazy execution, those dataframes you do not need cannot be cleared unless you do use them (why define them otherwise?).So when doing an action, spark will execute all code that is necessary.  If you run into memory issues, you can do...

  • 0 kudos
alesventus
by • Contributor
  • 2228 Views
  • 0 replies
  • 0 kudos

Performance issue: Running 50 notebooks from ADF

I have process in Data factory, that loads CDC changes from sql server and then trigger notebook with merge to bronze and silver zone. Single notebook takes about 1 minute to run but when all 50 notebooks are fired at once the whole process takes 25 ...

Data Engineering
performance issue
Reply
  • 2228 Views
  • 0 replies
  • 0 kudos
Greg
by • New Contributor III
  • 2870 Views
  • 1 replies
  • 4 kudos

How to reduce storage space consumed by delta with many updates

I have 1 delta table that I continuously append events into, and a 2nd delta table that I continuously merge into (streamed from the 1st table) that has unique ID's where properties are updated from the events (An ID represents a unique thing that ge...

Data Engineering
Reply
  • 2870 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Jb11
New Contributor II
  • 4 kudos

Did you already solved this problem?

  • 4 kudos
bfridley
by • New Contributor II
  • 5302 Views
  • 2 replies
  • 0 kudos

DLT Pipeline Out Of Memory Errors

I have a DLT pipeline that has been running for weeks. Now, trying to rerun the pipeline with the same code and same data fails. I've even tried updating the compute on the cluster to about 3x of what was previously working and it still fails with ou...

bfridley_1-1695328329708.png bfridley_2-1695328372419.png
Data Engineering
Reply
  • 5302 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
rajib_bahar_ptg
New Contributor III
  • 0 kudos

I'd focus on understanding the codebase first. It'll help you decide what logic or data asset to keep or not keep when you try to optimize it. If you share the architecture of the application, the problem it solves, and some sample code here, it'll h...

  • 0 kudos
1 More Replies
gkrilis
by • New Contributor
  • 11761 Views
  • 1 replies
  • 0 kudos

How to stop SparkSession within notebook without errr

I want to run an ETL job and when the job ends I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling spark.stop() the job returns with status failed even though it has f...

Data Engineering
cluster
SparkSession
Reply
  • 11761 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
PremadasV
New Contributor II
  • 0 kudos

Please refer to this Job fails, but Apache Spark tasks finish - Databricks

  • 0 kudos
Gilg
by • Contributor II
  • 1649 Views
  • 0 replies
  • 0 kudos

Add data manually to DLT

Hi Team,Is there a way that we can add data manually to the tables that are generated by DLT?We have done a PoC using DLT for Sep 15 to current data. Now, that they are happy, they wanted the previous data from Synapse and put into Databricks.I can e...

Data Engineering
Reply
  • 1649 Views
  • 0 replies
  • 0 kudos
Martin1
by • New Contributor II
  • 12033 Views
  • 2 replies
  • 1 kudos

Referring to Azure Keyvault secrets in spark config

Hi allIn spark config for a cluster, it works well to refer to a Azure Keyvault secret in the "value" part of the name/value combo on a config row/setting.For example, this works fine (I've removed the string that is our specific storage account name...

Data Engineering
Reply
  • 12033 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
kp12
New Contributor II
  • 1 kudos

Hello,Is there any update on this issue please? Databricks no longer recommend mounting external location, so the other way to access Azure storage is to use spark config as mentioned in this document - https://learn.microsoft.com/en-us/azure/databri...

  • 1 kudos
1 More Replies
scrimpton
by • New Contributor II
  • 5346 Views
  • 0 replies
  • 0 kudos

Permanently delete dropped table (Unity Catalog)

The recommendation before dropping a table is to do a DELETE then VACUUM RETENTION 0 (recommended in DEV).If you DROP the table without doing a DELETE|VACUUM, your table will be soft deleted with your entire data (permanently deletedin 30 days) and y...

Data Engineering
Reply
  • 5346 Views
  • 0 replies
  • 0 kudos
scrimpton
by • New Contributor II
  • 2980 Views
  • 0 replies
  • 0 kudos

Statistics for rearranged columns

The table property dataSkippingNumIndexedCols that gets statistics for a table starts from left to right. I am wondering what will happen to the statistics for both new and old records if we add a column in between using FIRST|AFTER identifier. 

Vince_03_0-1696039970438.png
Data Engineering
Reply
  • 2980 Views
  • 0 replies
  • 0 kudos
marvin1
by • New Contributor III
  • 865 Views
  • 0 replies
  • 0 kudos

Bamboolib error

What is the status of bamboolib?  I understand that it is public preview but I'm unable to find any support references.  I am getting error below.  I've tried installing in a notebook, on a cluster, creating a pandas dataframe and running bam, etc.  ...

Data Engineering
Reply
  • 865 Views
  • 0 replies
  • 0 kudos
938452
by • New Contributor III
  • 9149 Views
  • 0 replies
  • 0 kudos

Spark is not reading Kinesis Data as fast as specified

Hi Databricks community team,I have code as below"""df = spark.readStream \.format("kinesis") \.option("endpointUrl", endpoint_url) \.option("streamName", stream_name) \.option("initialPosition", "latest") \.option("consumerMode", "efo") \.option("ma...

Data Engineering
Reply
  • 9149 Views
  • 0 replies
  • 0 kudos
Labels