cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

AWS Databricks- Out of Memory issue in Delta live tables

Yaadhudbe
New Contributor

Thursday

I have been using Delta live tables more than a year and have implemented good number of DLT pipelines ingesting the data from S3 bucket using the SQS. One of my pipelines process large volume of data. The DLT pipeline reads the data using CloudFiles scripted in SQL language. It is straight forward without any inferSchemaType true. So its accepts all the source data as is into the RAW layer. As part of performance test, we are loading large volume of data and the Worker and driver are configured with 32GB 16 core with worker nodes ranging from 2  to 12. While ingesting the large volume of data, the pipeline never failed but used to take long time to process the data. But recently, it started throwing OOM error while ingesting 500K records with each file size less 100KB.

Yaadhudbe_0-1733411659495.png

Our pipeline is running in PREVIEW mode with Unity catalog implemented. Let me know if any has any solution..

0 Kudos
1 REPLY 1

Alberto_Umana
Databricks Employee
Databricks Employee

Thursday

Hi @Yaadhudbe,

We would need to review your DLT setup, cluster settings and spark processing to better understand the OOM errors and possible suggestions to mitigate the issue.

I suggest to file a case with us to conduct a proper investigation.

 https://docs.databricks.com/en/resources/support.html

0 Kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Announcements