Open-source app on Databricks system tables - looking for people to build it with me

[Hi3]

Hi,

I have been working with Databricks for a while and I kept rebuilding the same cost checks on system tables. I put them into a small open-source Streamlit app: spend, avoidable cost, and what to fix, from billing, access, compute, lakeflow, and query.

You can try it easily :

pip install -r app/requirements.txt
streamlit run app/app.py

Repo: https://github.com/HiGrp/05-Databricks-FinOps-App

I would rather build this with other Databricks users than on my own. If you care about cost, system tables, or FinOps and want to join you are welcome.

[AbdessamadSb]

Really nice project this looks genuinely useful for anyone working with Databricks costs and system tables I’ll take some time to go through the repo and see where I can contribute or help improve things

[Armanillo_RoK]

While I'm curious about your solution I need to check a few items I had on my radar already: 

1) Governance Hub - Azure Databricks | Microsoft Learn this might be what my clients (enterprises) have been looking for in Databricks (it now says it takes up to 12 hours to prepare the costs insights, so I just need to give it a little time). It's currently in Beta, however IMHO this is will reach GA relatively soon, due to the demand.

2) Then there's also GitHub - databricks-solutions/cloud-infra-costs · GitHub so I'll take a look and play with it as well

3)FinOpsWay - FinOps for Databricks | Track Databricks Costs this seems to provide an interesting view about the costs, however for corporate using such an extension wouldn't get through due to security reasons likely.

Thanks and let me see about the contribution. 

[AbdessamadSb]

Yeah that’s true. Bigger companies can be very cautious when it comes to external tools, especially because of security and compliance requirements.
It would be really nice if there was a proper app like this directly inside Databricks. I think that would make adoption much easier for enterprise users while still giving them the cost visibility they need.

[Hi3]

Thanks for the feedbacks very relevant.

One thing I should clarify about the project: the GitHub code can be deployed as a Streamlit app directly inside a customer's Databricks workspace. The idea is to keep the cost analysis close to the system tables and avoid sending the data to an external service.

I also tried to publish it as a Databricks Marketplace app, but it wasn't accepted. So I decided to share the project openly with the community instead.

If there is enough interest, I'd be happy to evolve it into something that can be deployed more easily inside enterprise Databricks environments, especially around security, governance and deployment.