Delta Sharing Egress Pipeline for Azure

[Elben]

We are currently investigating the options for implementing a multi tenancy solution where clients are separated but share data using delta sharing. Are there any way to track cost for reading data in Azure. It seems like Delta Sharing Egress Pipeline - Databricks Marketplace only supports an AWS setup.

Any help on this is appreciated

[BS_THE_ANALYST]

@Elben there's alot of billing tables you can get access via system tables. Check the docs out:
https://learn.microsoft.com/en-us/azure/databricks/admin/usage/system-tables 
https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/jobs-cost 

There's also Usage Dashboards you can plug straight into your Databricks Environment to give you a running start: https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage 

If you haven't already, it'd definitely be worth using Tags https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags 

All the best,
BS

[Elben]

Thanks. I'm aware of the very neat features for analyzing cost in databricks, but we are also interested in monitoring the cost for the underlying storage and network. It seems that this is indeed possible in AWS using s3, but not supported in Azure. 

[BS_THE_ANALYST]

Hmm, 
@Elben I found this in the documentation: https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress 
Delta Sharing + manage-egress

I'll keep having a little dig around. 

All the best,
BS

 

[BS_THE_ANALYST]

@Elben 
Seems like this part has information around the pricings: https://www.databricks.com/sites/default/files/2024-08/databricks-data-transfer-pricing.pdf 

After some research, I think your route may be to leverage Azure Cost Management, within Azure (given that the databricks system tables don't suffice).
https://learn.microsoft.com/en-ie/answers/questions/2133485/azure-databricks-usage-cost 

There's like an API which you could hit or leverage the SDK: https://learn.microsoft.com/en-us/python/api/overview/azure/mgmt-costmanagement-readme?view=azure-py... , I would lean more towards these routes for automation. However, to walk before you run, it might be worth seeing if you can download extracts manually using Azure Cost Management and seeing if they match your requirement. I'm assuming this should capture what you're after. The databricks bill, within the Azure Cost Management, likely won't be split out as granular as it would be from within Databricks System Tables, but I assume you could augment the two. 

All the best,
BS