Unable to load historical data via DLT workflow

[Rpabs]

Hi

I'm trying to load historical data with the DLT workflow and facing a Out of Memory or Executor heartbeat lost error. The historical data loads fine with normal processing, but fails within DLT workflow. Tried repartitioning and scaling cluster, but no success.

[Kaniz]

Hi @Rpabs, Steps to determine if OOM error is the reason for executor failure:  

1. Open Spark UI  

2. Click on "Stages"  

3. Click on "Failed stages"  

4. Click on the description of the failed stage  

5. Revthe the bottom of the stage details page  

6. Sort the list of tasks on the error column to check for the OOM error message

• Solutions to resolve executor failures due to OOM:  

1. Check for data skew in distribution across executors  

2. Use Adaptive Query Execution (AQE) to detect and resolve data skew automatically 

 3. Consider edge cases where AQE may fail to resolve data skew