Hi everyone!
I’m running a config-driven ingestion stack that uses the Databricks SDK (Python notebooks + GitHub Actions). All logging currently uses the standard Python logging module inside notebooks/jobs (example: ingest.py, logger.py).

I’d like to move beyond storing log files in ADLS and instead push every run’s logs into Azure Monitor / Log Analytics so ops teams can query KQL, build alerts, etc. I know I can enable Databricks “Diagnostic settings” to pipe cluster/job/system logs into a Log Analytics workspace, but I’m trying to understand the cleanest way to integrate our custom Python logger output as well. Ideally I’d keep the current logging API but add the necessary handlers/exports so everything lands in a single Log Analytics workspace alongside Databricks diagnostics.

Does anyone have a reference architecture or step-by-step instructions for:

Configuring Databricks workspace diagnostics to Log Analytics (which categories matter for notebooks/jobs)?
Wiring Python logging inside notebooks/jobs so custom logs appear in Log Analytics (e.g., via OpenCensus/opencensus-ext-azure, Data Collection API, or a recommended Databricks pattern)?
Managing this through Infrastructure-as-Code or CI/CD (we already store configs under config/*.yaml and run automation via workflows)
If there’s an official Databricks/Microsoft doc or sample repo that shows a full pipeline (notebook logs + diagnostics + Log Analytics queries), I’d really appreciate pointers. Thanks!   # Python #logging, #Azure Monitor #Log Analytics