Are you using Unity Catalog? Because you could leverage the automatic lineage in this case. It shows everything that comes downstream from this silver layer, you could easily check which columns are being used and the lineage.
Hey @dpc
I don't know if you can do it without customizing a query parser... I don't have this exact script but maybe this could help. First step would be to find out which tables haven't been queried a lot in the past days (in this case, 90 days). ...
You can find some tips here: https://community.databricks.com/t5/technical-blog/top-10-query-performance-tuning-tips-for-databricks-serverless/ba-p/43218 And here: https://www.databricks.com/discover/pages/optimize-data-workloads-guide
My overall re...
An idea, not sure if it helps you... You could set up a new task dependent on every other task, but with the run option set to "Run If at least one dependency failed". This new task would be a simple API call to the CANCEL RUN or CANCEL ALL RUNS OF A...
Notebooks are stored in the client's cloud storage. The diagram might have confused you, but it just goes to show that you access the notebooks via control plane, not that they're stored there. Azure Databricks automatically creates the workspace sto...
