Best practice for logging in Databricks notebooks?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-02-2022 03:30 PM
What is the best practice for logging in Databricks notebooks?
I have a bunch of notebooks that run in parallel through a workflow. I would like to keep track of everything that happens such as errors coming from a stream. I would like these logs to be maintained somewhere either in DBFS or in a storage account.
I got the built-in logging module working but you have to manually transfer the log file from a temp folder in file: to dbfs:/FileStore/log_folder/text.log. DBFS throws an error if the log file is directly assigned to its path with the FileHandler.
This basically works for my purposes but what is the actual best practice of doing it in Databricks?
- Labels:
-
Best practice
-
Databricks Notebooks
-
Logging
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-02-2022 11:14 PM
Configuring verbose audit logs and configuring audit log delivery can be one of the best practises.
https://docs.databricks.com/administration-guide/account-settings/audit-logs.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-03-2022 02:09 AM
Please consider integration of databricks with datadog https://www.datadoghq.com/blog/databricks-monitoring-datadog/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-03-2022 07:42 AM
@Gimwell Young AS @Debayan Mukherjee mentioned if you configure verbose logging in workspace level, logs will be moved to your storage bucket that you have provided during configuration. from there you can pull logs into any of your licensed log monitoring tool like eg: Splunk etc. also same config can be used to monitor unity catalog logs. as @Hubert Dudek mentioned if you configure datadog, you will have graphical view of resources that are being consumed by workspace like no of clusters active, jobs active etc...