Reporting serverless costs to azure costs

[asharkman]

So, we've just recently applied serverless budget polices to some of our vector searches and apps. 
At the moment they're all going to azure under one general tag that we created.

However, we needed more definition. So i added the serverless budget policies. I can see my tags in the billing.usage table.
However, these don't seem to be feeding up to azure? Is there a way to do that? Are they appearing under something else? How can i find them?
I'm currently exporting the costs from azure and reporting from that. 

Cheers.

[SP_6721]

Hi @asharkman ,

Tags from serverless budget policies are included in Azure cost exports.
Try checking in the Azure Portal under Cost Analysis, add Tag as a filter, and enter your custom tag key.
Also it can take a few hours for the tags to appear in Azure billing exports.

https://learn.microsoft.com/en-us/azure/databricks/admin/usage/budget-policies

[asharkman]

 

  

Hi Thanks for that. 
So, yea, i went to the azure cost management section and am exporting the costs. However, the newly added serverless budget tags are not appearing in the exports. 
Neither are they appearing on the dashboard. I've tried filtering the dashboard on the tags.
Oddly, of the billing origin product, "model serving" is filtering through, but "vector search" and "apps" are not. 
As in i can see model serving in azure cost management, but not vector search or apps.

Cheers

[Bhaskar27]

Just to note, tags from budget policies only apply to new usage after the policy is created, so they won’t show up on any past costs.
If that’s already the case and the tags still aren’t appearing, it might be best to reach out to Databricks or Azure Support to look into it further.

[asharkman]

Hi, thanks for replying, yea we applied the tags on the 9th and it's been over a week and only some of them are filtering through to azure, I've reached out to MS using a support ticket.

I'll raise something with databricks and see if that goes anywhere.
Cheers.

[johnsonjame]

You might want to double-check if the tags from your Databricks serverless budget policies are being properly propagated to the Azure cost management system. Azure Cost Management typically only recognizes tags that are applied at the resource level within Azure. Since Databricks serverless is a managed service, those internal tags may not automatically map to Azure-native tags. You could try using Azure Cost Allocation Rules or Azure Tag Inheritance (if available for your setup) to help align costs with your custom tags. Another option might be to script tag mapping during your export/reporting workflow to manually associate those values.

https://learn.microsoft.com/en-us/azure/databricks/admin/usage/budget-policies

[asharkman]

Hi, thanks for this. 
it doesn't seem to be the simple straightforward solution i thought it was going to be. 
They've advertised serverless budget polices as a way to monitor costs, but unfortunately, it seems like you can only monitor them within Databricks. which is kind of useless when everything else is being monitored in azure!

I'll have a look at Azure Cost Allocation Rules or Azure Tag Inheritance and see how they pan out. 

When you say "script tag mapping during your export/reporting workflow to manually associate those values." how would that work? 

Cheers

Paul

[asharkman · Accepted solution]

OK, so raised it with MS, they raised it with Databricks, turns out it's behaving as expected. Serverless billing isn't enabled on vector searches or apps. 

However, there is a plan to add them to the billing process. This piece of work is planned to be complete in Sept/Oct.

[mrsimon0007]

Billing or set up explicit export pipelines. Check whether your serverless budget policy tags are under a different namespace in Azure, as sometimes they show up nested.