https://community.databricks.com/t5/administration-architecture/determining-spill-from-system-tables/m-p/104165#M2673 <P>I'm trying to optimize machine selection (D, E, or L types on Azure) for job clusters and all-purpose compute and am struggling to identify where performance is sagging on account of disk spill.&nbsp; Disk spill would suggest that more memory is needed.&nbsp; I can get there from the Spark UI but am looking for historical diagnostics.</P><P>As of January 2025, system.compute.node_timeline is telling me useful things but not spill, explicitly.</P><P><A href="https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline-table-schema" target="_blank" rel="noopener">https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline-table-schema</A></P><P>Help appreciated.</P> Fri, 03 Jan 2025 23:37:40 GMT drumcircle 2025-01-03T23:37:40Z https://community.databricks.com/t5/administration-architecture/determining-spill-from-system-tables/m-p/104165#M2673 <P>I'm trying to optimize machine selection (D, E, or L types on Azure) for job clusters and all-purpose compute and am struggling to identify where performance is sagging on account of disk spill.&nbsp; Disk spill would suggest that more memory is needed.&nbsp; I can get there from the Spark UI but am looking for historical diagnostics.</P><P>As of January 2025, system.compute.node_timeline is telling me useful things but not spill, explicitly.</P><P><A href="https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline-table-schema" target="_blank" rel="noopener">https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline-table-schema</A></P><P>Help appreciated.</P> Fri, 03 Jan 2025 23:37:40 GMT https://community.databricks.com/t5/administration-architecture/determining-spill-from-system-tables/m-p/104165#M2673 drumcircle 2025-01-03T23:37:40Z https://community.databricks.com/t5/administration-architecture/determining-spill-from-system-tables/m-p/104167#M2674 <P>For historical diagnostics, you might need to consider setting up a custom logging mechanism that captures these metrics over time and stores them in a persistent storage solution, such as a database or a logging service. This way, you can query and analyze historical performance data, including disk spill, at any point in the future.</P> Sat, 04 Jan 2025 00:42:54 GMT https://community.databricks.com/t5/administration-architecture/determining-spill-from-system-tables/m-p/104167#M2674 Walter_C 2025-01-04T00:42:54Z