Hey!

You can approach this by analyzing cluster usage through Spark UI, monitoring DBU consumption, and leveraging cloud provider cost tracking.

Spark UI provides insights into job execution times, resource utilization, and idle vs. active periods. Reviewing these metrics can help identify inefficiencies and periods of underutilization. Additionally, Databricks charges DBUs (Databricks Units) based on cluster uptime and resource allocation, meaning costs accrue even when the cluster is idle. Factors like auto-scaling, instance types, and idle time can influence DBU consumption.

Another useful approach is checking the monitoring tools provided by your cloud provider. AWS CloudWatch, Azure Cost Management, and GCP Billing Reports can help you track instance uptime and correlate it with Databricks usage. This allows you to better understand how much you’re paying for actual workloads versus idle time.

If you’re using SQL clusters, Databricks offers built-in monitoring where you can track cluster uptime, under the SQL > Monitoring tab, the number of clusters started, tasks executed, and queued workloads. This can provide a clearer picture of usage patterns and help with cost optimization.

To optimize costs, consider enabling auto-termination to shut down clusters when they’re idle, using System Tables to analyze job-level costs, and possibly running workloads as scheduled jobs instead of interactive sessions when appropriate.

If you find this answer helpful, feel free to mark it as resolved or give it a 👍!

🙂