https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24092#M16715 <P>What is the best practice for logging in Databricks notebooks? </P><P></P><P>I have a bunch of notebooks that run in parallel through a workflow. I would like to keep track of everything that happens such as errors coming from a stream. I would like these logs to be maintained somewhere either in DBFS or in a storage account.</P><P></P><P>I got the built-in logging module working but you have to manually transfer the log file from a temp folder in <I>file: </I>to dbfs:/FileStore/log_folder/text.log. DBFS throws an error if the log file is directly assigned to its path with the FileHandler.</P><P></P><P>This basically works for my purposes but what is the actual best practice of doing it in Databricks?</P> Wed, 02 Nov 2022 22:30:05 GMT Gim 2022-11-02T22:30:05Z https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24092#M16715 <P>What is the best practice for logging in Databricks notebooks? </P><P></P><P>I have a bunch of notebooks that run in parallel through a workflow. I would like to keep track of everything that happens such as errors coming from a stream. I would like these logs to be maintained somewhere either in DBFS or in a storage account.</P><P></P><P>I got the built-in logging module working but you have to manually transfer the log file from a temp folder in <I>file: </I>to dbfs:/FileStore/log_folder/text.log. DBFS throws an error if the log file is directly assigned to its path with the FileHandler.</P><P></P><P>This basically works for my purposes but what is the actual best practice of doing it in Databricks?</P> Wed, 02 Nov 2022 22:30:05 GMT https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24092#M16715 Gim 2022-11-02T22:30:05Z https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24094#M16717 <P>Please consider integration of databricks with datadog <A href="https://www.datadoghq.com/blog/databricks-monitoring-datadog/" target="test_blank">https://www.datadoghq.com/blog/databricks-monitoring-datadog/</A></P> Thu, 03 Nov 2022 09:09:21 GMT https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24094#M16717 Hubert-Dudek 2022-11-03T09:09:21Z https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24096#M16719 <P>@Gimwell Young​&nbsp;AS @Debayan Mukherjee​&nbsp;mentioned if you configure verbose logging in workspace level, logs will be moved to your storage bucket that you have provided during configuration. from there you can pull logs into any of your licensed log monitoring tool like eg: Splunk etc. also same config can be used to monitor unity catalog logs. as @Hubert Dudek​&nbsp;mentioned if you configure datadog, you will have graphical view of resources that are being consumed by workspace like no of clusters active, jobs active etc...</P> Thu, 03 Nov 2022 14:42:35 GMT https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24096#M16719 karthik_p 2022-11-03T14:42:35Z https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24093#M16716 <P>Configuring verbose audit logs and configuring audit log delivery can be one of the best practises. </P><P><A href="https://docs.databricks.com/administration-guide/account-settings/audit-logs.html" alt="https://docs.databricks.com/administration-guide/account-settings/audit-logs.html" target="_blank">https://docs.databricks.com/administration-guide/account-settings/audit-logs.html</A></P><P></P> Thu, 03 Nov 2022 06:14:33 GMT https://community.databricks.com/t5/data-engineering/best-practice-for-logging-in-databricks-notebooks/m-p/24093#M16716 Debayan 2022-11-03T06:14:33Z