java.lang.OutOfMemoryError on Data Ingestion and Storage Pipeline

[hv129]

I have around 25GBs of data in my Azure storage. I am performing data ingestion using Autoloader in databricks. Below are the steps I am performing:

1. Setting the enableChangeDataFeed as true.
2. Reading the complete raw data using readStream.
3. Writing as delta table using writeStream to the azure blob storage.
4. Reading the change feed data of this delta table using spark.read.format("delta").option("readChangeFeed", "true")...
5. Performing operations on the change feed table using withColumn (Performing operations on content column as well, might be taking a lot of computation).

Now I am trying to save this computed pyspark dataframe to my catalog but getting the error: java.lang.OutOfMemoryError. My databricks cluster has 1 Driver with 16GBs of memory and 4 nodes, max of 10 Workers with 16GBs of memory and 4 nodes each.

Is there a need to include more resources to the cluster or is there some way to optimize or replace the current pipeline?

[Kaniz]

Hi @hv129, The java.lang.OutOfMemoryError error typically occurs when the JVM heap space is exhausted. This can happen due to a variety of reasons such as large data sets, high computation requirements, ....

Here are some suggestions to optimize your current pipeline and potentially resolve the issue:

Optimize Data Workloads: Databricks, Apache Spark™, and Delta Lake have best practices and optimizat....

Simplify Data Ingestion: Utilize Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Li....

Optimize Writes to Delta Tables: Reduce latency when reading by running Auto Optimize to automatical....

Automate ETL Processing: Use Databricks workflows and jobs to author, manage, and orchestrate your s....

Increase Resources: If the issue persists, you might need to increase the resources of your Databric.... This could involve adding more nodes to the cluster or increasing the memory of the driver node.

Tune Spark Parameters: Adjusting Spark parameters such as spark.executor.memory, spark.driver.memory....

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!