topic Does Databricks have alerts / thresholds in place for cost monitoring? in Data Engineering

Does Databricks have alerts / thresholds in place for cost monitoring?

Anonymous — Wed, 16 Jun 2021 21:01:38 GMT

Re: Does Databricks have alerts / thresholds in place for cost monitoring?

sajith_appukutt — Thu, 17 Jun 2021 23:11:41 GMT

As a Databricks account owner or account admin, you can configure daily delivery of billable usage logs in CSV file format to an AWS S3 storage bucket, where you can make the data available for further analytics.

A simple pipeline for configuring budget thresholds and alerting could look like this

Configure billable usage logs delivery to an s3 bucket.
Configure autoloader to load this to delta table
Configure dashboards in Databricks SQL for reporting
Configure alerts in Databricks SQL by scheduling a query that checks a threshold

Re: Does Databricks have alerts / thresholds in place for cost monitoring?

aladda — Wed, 23 Jun 2021 04:00:31 GMT

You can also use tags to setup a chargeback mechanism within your organization for distributed billing - https://docs.databricks.com/administration-guide/account-settings/usage-detail-tags-aws.html