Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
Ok how many time was the cluster running? Sometimes depending on the time of the run there is no data populated to the metrics tab so probably wont be on the system table.
Mostly this issue occurs in Job Compute clusters. It runs 3 times a day. Sometimes, driver details not available and sometimes both worker and driver data not available.
Adding to my previous reply, during the same job run, i also use another job compute with single worker to run another task, this task completes in 6 minutes. For this job compute i get only worker data in system table.
Yes seems that the runs are 1 minute of execution so this might be the reason why the metrics are not loaded
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
