cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

[FinOps] Tagging queries in databricks

Etyr
Contributor

Hello,

I see that it is possible to tag catalogs/databases/tables. But I did not find a way to tag a query for our finop use case.

In Azure you can check billings dependings on tags.

A concrete example: 

In Azure Machine Learning, I have a schedule that will launch a Compute Cluster (tagged), where it will launch python code and connect to databricks SQL warehouse and Spark Cluster. The SQL Warehouse/spark clusters are commun to multiple projects. We do not want to assign 1 cluster per project since multiple projects access the same databases/tables. The python code can also access CosmosDB (tagged), and so on.

In this case, we do not know how much the query on databricks for this project has cost. We will knows for every other Azure ressource but not for Databricks.

How can we acheive this ?

I know we can add tags in a SparkSession, but Azure will not have a view on this, and it will not work on SQL Warehouses

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @Etyr, Thank you for reaching out with your question about tagging queries in Databricks. I understand the importance of the issue and I'm here to help you resolve it.

Please refer to the official documentation on monitoring usage using tags in Azure Databricks.

 

 

Etyr
Contributor

Thank you for your reply.

So I will have to tag the workspace + cluster to make it work. And if I want to seperate 100% the cost of 2 teams, I guess I'll have to make seperate clusters with different tags. I wish we could have tag queries and get the cost of a query depending on the cluster DBU/VM, but I think it doesn't exists. 

 

Etyr
Contributor

@yoav Hello, sorry I am not interrested in a payed solution.