topic Re: Delta Live table getting error of garbage collection after running few days in Data Engineering https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67624#M33389 <P>could you share the ganglia metrics&nbsp; and how size/type is your driver?</P> Mon, 29 Apr 2024 21:55:02 GMT jose_gonzalez 2024-04-29T21:55:02Z Delta Live table getting error of garbage collection after running few days https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67622#M33387 <P>Hi, i am using delta live table in continuous mode for a real time streaming data pipeline. After running the pipeline like 2-3 days i am getting this garbage collection error:</P><P>Driver/10.15.0.73 paused the JVM process 68 seconds during the past 120 seconds (57.32%) because of GC. We observed 5 such issue(s) since 2024-04-29T15:41:29.874Z.</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Darian_0-1714426883477.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7311iBE4805275ADBBB9D/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Darian_0-1714426883477.png" alt="Darian_0-1714426883477.png" /></span></P><P>Here are my cluster setting:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Darian_1-1714426964675.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7312iC6CAE9186190CFC4/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Darian_1-1714426964675.png" alt="Darian_1-1714426964675.png" /></span></P><P>&nbsp;</P><P>&nbsp;</P> Mon, 29 Apr 2024 21:43:31 GMT https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67622#M33387 Darian 2024-04-29T21:43:31Z Re: Delta Live table getting error of garbage collection after running few days https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67624#M33389 <P>could you share the ganglia metrics&nbsp; and how size/type is your driver?</P> Mon, 29 Apr 2024 21:55:02 GMT https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67624#M33389 jose_gonzalez 2024-04-29T21:55:02Z Re: Delta Live table getting error of garbage collection after running few days https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67723#M33425 <P>Here are the metrics:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Darian_1-1714494143681.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7337i0EBC7FE488EDE918/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Darian_1-1714494143681.png" alt="Darian_1-1714494143681.png" /></span></P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Darian_2-1714494193135.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7338iDEDDC9F5600ABB35/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Darian_2-1714494193135.png" alt="Darian_2-1714494193135.png" /></span></P><P>The size/type:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Darian_3-1714494245085.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7339i189595C3C6A1260B/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Darian_3-1714494245085.png" alt="Darian_3-1714494245085.png" /></span></P><P>Thanks!</P><P>&nbsp;</P><P>&nbsp;</P><P>&nbsp;</P> Tue, 30 Apr 2024 16:24:21 GMT https://community.databricks.com/t5/data-engineering/delta-live-table-getting-error-of-garbage-collection-after/m-p/67723#M33425 Darian 2024-04-30T16:24:21Z