topic Re: Databricks job keep getting failed due to GC issue in Data Engineering

Databricks job keep getting failed due to GC issue

shahabm — Fri, 02 Aug 2024 21:25:47 GMT

There is a job that running successful but it's for more than a month we are experiencing long run which gets failed. In the stdout log file(attached), there are numerous following messages:

[GC (Allocation Failure) [PSYoungGen:...] and [Full GC (System.gc()) [PSYoungGen:...]

It seems I am getting GC issues that take a longer time to run and then it fails every time. In one of the executors log within SparkUI\Executors page I see an error message (ExecLossReason.png) showing that "Executor decommission: worker decommissioned because of kill request from HTTP endpoint (data migration disabled)"

Then within Spark config parameters I added the following

spark.databricks.dataMigration.enabled true

I tried to use stronger Compute/Worker/Driver type but still I get the same failure message.

Any thoughts? How can I resolve this issue while the pipeline job is working correctly in DEV, UAT up to PROD but in QA?

Re: Databricks job keep getting failed due to GC issue

shahabm — Thu, 22 Aug 2024 00:53:01 GMT

Hi @Retired_mod

Your advice worked pretty fine and I could get rid of [GC (Allocation Failure) [PSYoungGen:...] totally and also by picking stronger driver/worker types, the issue in production went away.

I understood the default setting for GC was 'Parallel GC' and by configuring G1GC I can see more balanced behavior for the GC and also driver/workers are working more efficiently into some extent.

Thanks again,

Shahab

Re: Databricks job keep getting failed due to GC issue

siddhu30 — Fri, 22 Nov 2024 02:17:08 GMT

Hi @shahabm , I'm facing exactly the same issue and increasing driver type or number of workers isn't helping too. Could you please guide me how it got resolved for you as I don't see the comment or post in which you got advice. This problem causing so much delays and escalations in delivery. Appreciate your timely guidance on it.

Thanks in advance!

Regards,

Sid

Re: Databricks job keep getting failed due to GC issue

shahabm — Fri, 22 Nov 2024 14:12:36 GMT

Hi Sid,

These are the list of action items that helped me resolve the issue:

Change of GC algorithm into G1GC. It uses less resources and is more efficient.
Archived old ingested files that decreased the workload.
Chose a more stronger cluster with a more matching Databricks runtime version : 11.3 LTS (Spark 3.3.0, Scala 2.12)
chose a reasonable number of workers as min (1) and max (3) based on the log
tried to resolve all warning messages I could see in the log.

By these actions, the first step of my workflow ran but I had a separate issue with the next step which was a file system issue. It could not find some of the delta tables and the locations they were put in. I could resolve the issue this way that let me complete the job.

Mounted the unmounted file system by comparing them to a working environment. I used dbutils to show me the mounted points. There was a misleading mounted location. I could remove it. The issue resolved.
I ran few performance improvements on my delta tables afterward

Re: Databricks job keep getting failed due to GC issue

siddhu30 — Fri, 22 Nov 2024 14:56:17 GMT

Thanks a lot @shahabm for your prompt response, appreciate it. I'll try to debug in this direction.

Thanks again!

Re: Databricks job keep getting failed due to GC issue

isyed — Tue, 04 Feb 2025 07:26:53 GMT

Hi Sid,

Is this issue resolved? we are also experiencing the same just wanted to know if the above steps helped in resolving it and any additional steps you have followed to resolve your issue.

Thanks

Re: Databricks job keep getting failed due to GC issue

siddhu30 — Tue, 24 Feb 2026 15:16:06 GMT

Hi isyed,

Apologies for the late response.

For our use case, we tried to change the code from pyspark dataframes to spark sql, which instead of keeping all the records into the memory, writes to the tables and then perform next loop. Ours is typical hierarchical looping done over 200 million of records. Every loop used to store in dataframe and then calculate the next hierarchy and so on, which caused the issue. After changing the logic to SQL (Insert records into temp tables every loop), the code was running faster since there's no storing data in memory as every loop, the records are written into temp tables and next loop record count is reduced.

Hope this helps.