tagging for serverless compute

[Charansai]

Hi,

I am working with Databricks Serverless Compute and All-Purpose Compute and have several questions regarding tagging, system tables, and budget policies. Specifically, I need clarification on how these work together in Databricks, and how I can manage them using Terraform.

1. How do I tag resources in Databricks?

   • What are the best practices for tagging both Serverless Compute and All-Purpose Compute resources?

   • Can I use common tags like Environment, Owner, CostCenter, Project, etc., for both types of compute?

   • How can I implement these tags for resource management, especially for budgeting purposes?

2. Can I manage these tags with Terraform?

   • Is it possible to tag Serverless Compute and All-Purpose Compute resources through Terraform?

   • How can I automate the tagging process for both types of compute resources?

3. How do System Tables work in the real world in Databricks?

   • What are the typical use cases for system tables in Databricks?

   • How can system tables be managed and accessed for monitoring, auditing, or troubleshooting in production environments?

4. How can I configure Budget Policies in Databricks?

   • What is the best way to set up a budget policy in Databricks for cost control?

   • How can I associate budget policies with resources like Serverless Compute and All-Purpose Compute using tags?

5. Can these policies and tags be managed through Terraform?

   • How do I configure budget policies and tags for Serverless Compute and All-Purpose Compute using Terraform to ensure automated and consistent resource management?

Note: I am not referring to Serverless SQL Warehouse in this question.

Any insights, examples, or resources you can share would be greatly appreciated.

Thank you!

[szymon_dybczak]

Hi @Charansai ,

Databricks automatically adds some default tags to compute resources that provide some basic information like name, ID, and creator. You can use these tags to attribute the usage to Databricks.
The default tags automatically propagate to detailed cost analysis reports that you can access in the Azure portal.

So for example here you can see the list of all default tags that databricks add to compute:

You can also define custom tags. Custom tags let you attribute compute usage to specific teams, projects, or cost centers with more granularity than default tags. These tags are applied by users or admins and propagate to both your account's usage logs and applicable cloud resources. These tags are also used to create and monitor budgets in your Azure Databricks account.

You can apply custom tags to different resources like Workspace, Pool, All-purpose and job compute and SQL warehouse

To tag serverless compute workloads you need to use serverless budget policies. When a user is assigned a serverless budget policy, their serverless usage is automatically tagged with their policy's custom tags. Serverless budget policies can be applied to serverless notebooks, jobs, pipelines, and model serving endpoints

You can apply custom tags to compute using terraform. For example, cluster resource supports attribute custom_tags that you can use to apply your own set of tags.

databricks_cluster | Resources | databricks/databricks | Terraform | Terraform Registry

You can also create databricks budget policy via terraform. Budget policies consist of tags that are applied to any serverless compute activity incurred by a user assigned to the policy.

databricks_budget_policy | Resources | databricks/databricks | Terraform | Terraform Registry

Last, but not least. System tables are a Databricks-hosted analytical store of your account's operational data found in the system catalog. System tables can be used for historical observability across your account. They are used for multiple things like tracking lineage, auditing, tracking billable usage, tracking query history and many more. 

[Charansai]

I am now more inclined towards the serverless and tagging, would be more helpful if you could take a real time use case and elaborate on it. Ty

[szymon_dybczak]

Let's say that you have Finance department in your company and you want to track the cost of serverless compute generated by that particular department. First you need to create budget policy. To create budget policy, go to your databricks workspace and click settings (1):

Now, in Setting page click on compute (2) and then under Policies click manage (3) to configure Serverless budget policy:

Now, click create button (4):

Now, provide a name for your budget policy and all tags that should be part of that specific policy and click create:

Once you created serverless budget policy you can associate it with job/notebook/pipeline or serving endpoint which will allow you to track the cost. 
For instance, I've created a notebook that I'd like to run with serverless compute and track cost. Connect to serverless compute and then go to configuration:

Now, selected policy you want to apply (in my case Finance Department policy):

After a policy is applied to a notebook, job, or Lakeflow Declarative Pipelines, any tags contained in the policy propagate to your system.billing.usage system table in the custom_tags column.