Original post published on 30 May 2024, here.
FinOps and Databricks System Tables
Dear all,
I have to admit that FinOps and System Tables are not the most interesting topic to talk about it, but everything we need to put to production will generate a cost and it is important to monitor the usage, etc.
Having said that, I would like to talk about FinOps and System Tables.
Introduction to Databricks System Tables
System Tables are a Databricks-hosted analytical store for operational and usage data.
System Tables can be used for monitoring, analyzing performance, usage, and behavior of Databricks Platform components. By querying these tables, users can gain insights into how their jobs, notebooks, users, clusters, ML endpoints, and SQL warehouses are functioning and changing over time. This historical data can be used to optimize performance, troubleshoot issues, track usage patterns, and make data-driven decisions.
Overall, System Tables provide a means to enhance observability and gain valuable insights into the operational aspects of Databricks usage, enabling users to better understand and manage their workflows and resources.
Databricks System Tables offer a comprehensive solution for monitoring, analyzing, and optimizing the operational aspects of the Databricks platform. These tables are perfect for:
📊 Cost and Usage Analytics: Track and manage your expenses effectively.
🚀 Efficiency Analytics: Optimize performance and resource allocation.
🔒 Audit Analytics: Ensure compliance and security with comprehensive auditing.
📜 GDPR Regulation: Maintain data privacy and adhere to regulatory standards.
🎯 Service Level Objective Analytics: Meet your SLAs with precision.
✅ Data Quality Analytics: Ensure the integrity and quality of your data.
By querying System Tables, gain valuable insights into jobs, notebooks, clusters, ML endpoints, and SQL warehouses. Harness this historical data to troubleshoot issues, track usage patterns, and make informed, data-driven decisions.
Try it now:
Here we have a Demo where you can use System Tables in your Databricks workspace.
System Tables: Billing Forecast, Usage Analytics, and Access Auditing With Databricks Unity Catalog
https://www.databricks.com/resources/demos/tutorials/governance/system-tables
https://notebooks.databricks.com/demos/uc-04-system-tables/index.html#
Databricks Cost Analysis using Power BI Template
For the Power BI users, I would recommend this material
Cost control and cross charge with power bi template on top of Databricks system tables
https://community.databricks.com/t5/technical-blog/databricks-cost-analysis-and-cross-charge-with-po...
https://github.com/lipinght/Databricks-cost-analysis-using-powerbi
Learn more:
https://docs.databricks.com/en/admin/system-tables/index.html
https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/
https://github.com/drcaiomoreno/databricks-finops-system-tables
Databricks Specialist Sessions (Observability and FinOps with Overwatch and more): https://vimeo.com/869043852
UCX
If you do not have Unity Catalog enabled, you will need Unity Catalog to use System Tables, I recommend you to use UCX to start your Unity Catalog adoption:
https://github.com/databrickslabs/ucx
