Serverless Workspace Observability

[APJESK]

I’m setting up observability for a Databricks serverless workspace on AWS and need some guidance.
I know we can configure audit logs for S3 delivery, but I’m unsure if that alone is sufficient.

For a complete observability setup especially when integrating with CloudWatch, Splunk, or Kibana

• Do we only need to enable audit-log delivery to S3, or

• Are there other logs that should also be routed to S3 for best practices?

If anyone has implemented observability in a serverless Databricks workspace, I’d love to hear what log sources you included and any reference docs or patterns you followed.

[sarahbhord · Accepted solution]

Hey @APJESK - thanks for reaching out! 

For comprehensive observability in a Databricks serverless workspace on AWS, particularly when integrating with tools like CloudWatch, Splunk, or Kibana, enabling audit log delivery to S3 is a crucial first step, but it is not the only log source to consider. As you noted, it is a good idea to not rely solely on audit logs—external cloud logs help detect issues Databricks can’t see alone.

Logs you can route to S3:

- Databricks Audit Logs (you've got these): Enable delivery to S3 to capture detailed platform-level activity (user actions, resources, permissions).

- AWS Cloud-Native Logs: Include CloudTrail, S3 access logs, and VPC flow logs for visibility into cloud-level actions like authentication, data access, and network traffic.

- Job, Pipeline, and Query Logs: Monitor Databricks event logs (for jobs, pipelines, and SQL warehouse activity) using system tables or metrics endpoints for operational health and anomaly detection.

So, it is best practice to aggregate and monitor all these log types for comprehensive security and operational insight. You can integrate logs into SIEM or monitoring systems (CloudWatch, Splunk, Kibana) using ETL pipelines or native AWS integrations.

You can find more information in the docs for Operational Excellence.

I hope this is helpful!

Sarah