Hello,
I see that it is possible to tag catalogs/databases/tables. But I did not find a way to tag a query for our finop use case.
In Azure you can check billings dependings on tags.
A concrete example:
In Azure Machine Learning, I have a schedule that will launch a Compute Cluster (tagged), where it will launch python code and connect to databricks SQL warehouse and Spark Cluster. The SQL Warehouse/spark clusters are commun to multiple projects. We do not want to assign 1 cluster per project since multiple projects access the same databases/tables. The python code can also access CosmosDB (tagged), and so on.
In this case, we do not know how much the query on databricks for this project has cost. We will knows for every other Azure ressource but not for Databricks.
How can we acheive this ?
I know we can add tags in a SparkSession, but Azure will not have a view on this, and it will not work on SQL Warehouses
