[FinOps] Tagging queries in databricks

[Etyr]

Hello,

I see that it is possible to tag catalogs/databases/tables. But I did not find a way to tag a query for our finop use case.

In Azure you can check billings dependings on tags.

A concrete example: 

In Azure Machine Learning, I have a schedule that will launch a Compute Cluster (tagged), where it will launch python code and connect to databricks SQL warehouse and Spark Cluster. The SQL Warehouse/spark clusters are commun to multiple projects. We do not want to assign 1 cluster per project since multiple projects access the same databases/tables. The python code can also access CosmosDB (tagged), and so on.

In this case, we do not know how much the query on databricks for this project has cost. We will knows for every other Azure ressource but not for Databricks.

How can we acheive this ?

I know we can add tags in a SparkSession, but Azure will not have a view on this, and it will not work on SQL Warehouses

[Kaniz]

Hi @Etyr, Thank you for reaching out with your question about tagging queries in Databricks. I understand the importance of the issue and I'm here to help you resolve it.

Please refer to the official documentation on monitoring usage using tags in Azure Databricks.

[Etyr]

Thank you for your reply.

So I will have to tag the workspace + cluster to make it work. And if I want to seperate 100% the cost of 2 teams, I guess I'll have to make seperate clusters with different tags. I wish we could have tag queries and get the cost of a query depending on the cluster DBU/VM, but I think it doesn't exists. 

[Etyr]

@yoav Hello, sorry I am not interrested in a payed solution.