topic Re: [FinOps] Tagging queries in databricks in Data Engineering

[FinOps] Tagging queries in databricks

Etyr — Mon, 18 Mar 2024 10:04:21 GMT

Hello,

I see that it is possible to tag catalogs/databases/tables. But I did not find a way to tag a query for our finop use case.

In Azure you can check billings dependings on tags.

A concrete example:

In Azure Machine Learning, I have a schedule that will launch a Compute Cluster (tagged), where it will launch python code and connect to databricks SQL warehouse and Spark Cluster. The SQL Warehouse/spark clusters are commun to multiple projects. We do not want to assign 1 cluster per project since multiple projects access the same databases/tables. The python code can also access CosmosDB (tagged), and so on.

In this case, we do not know how much the query on databricks for this project has cost. We will knows for every other Azure ressource but not for Databricks.

How can we acheive this ?

I know we can add tags in a SparkSession, but Azure will not have a view on this, and it will not work on SQL Warehouses

Re: [FinOps] Tagging queries in databricks

Etyr — Thu, 21 Mar 2024 08:00:25 GMT

Thank you for your reply.

So I will have to tag the workspace + cluster to make it work. And if I want to seperate 100% the cost of 2 teams, I guess I'll have to make seperate clusters with different tags. I wish we could have tag queries and get the cost of a query depending on the cluster DBU/VM, but I think it doesn't exists.

Re: [FinOps] Tagging queries in databricks

Etyr — Mon, 25 Mar 2024 07:25:42 GMT

@yoav Hello, sorry I am not interrested in a payed solution.