https://community.databricks.com/t5/data-engineering/cluster-logs-missing/m-p/20462#M13803 <P>On the Databricks cluster UI, when I click on the Driver logs, sometimes I see historic logs and sometimes I see logs for the last few hours. Why do we see this inconsistency</P> Thu, 24 Jun 2021 18:10:13 GMT brickster_2018 2021-06-24T18:10:13Z https://community.databricks.com/t5/data-engineering/cluster-logs-missing/m-p/20462#M13803 <P>On the Databricks cluster UI, when I click on the Driver logs, sometimes I see historic logs and sometimes I see logs for the last few hours. Why do we see this inconsistency</P> Thu, 24 Jun 2021 18:10:13 GMT https://community.databricks.com/t5/data-engineering/cluster-logs-missing/m-p/20462#M13803 brickster_2018 2021-06-24T18:10:13Z https://community.databricks.com/t5/data-engineering/cluster-logs-missing/m-p/20463#M13804 <P>This is working per design! This is the expected behavior. </P><P></P><P>When the cluster is in terminated state, the logs are serviced by the Spark History server hosted on the Databricks control plane. </P><P>When the cluster is up and running the logs are serviced by the Spark Driver at that point in time. </P><P></P><P>Because of this architecture, when the cluster is in the terminated state you will see the logs for the last 30 days and when the cluster is up and running you will see the logs from the last restart/start of the cluster. </P> Thu, 24 Jun 2021 18:12:12 GMT https://community.databricks.com/t5/data-engineering/cluster-logs-missing/m-p/20463#M13804 brickster_2018 2021-06-24T18:12:12Z