Hi All,

I'm after some guidance on how to identify massive (100000%) spikes in bandwidth usage (and related costs) in the azure databricks provisioned/managed resource group storage account & stop them

These blips are adding 30-50% to our monthly costs. This is creating uncertainty in the organisation as we can't accurately predict our usage costs.

Our environment is still very much in the POC stage

I believe it's related to the geo-redundancy and databricks copying massive amounts of data to another region even when the clusters are turned off

I can't dig deeper because "system deny" security settings created by databtricks when provisioning the app lock me out even though I'm the subscription admin

(Azure support have not been helpful - just repeating that "yes, these costs were incurred by the databricks managed storage account")

This is weird because I use externally managed tables for everything linked to a storage account I control. I don't use DBFS for anything, especially not 1000s of gigs worth of data.

1.what is happening?

- the timestamps are on the weekend/overnight - our sql service and clusters are off outside business hours and are set to autostop & doesn't align to custers being turned on or any jobs - all of which are relatively simple and small

our 'raw data/datalake' is 2.9gig

the unity catalog + databricks tables are 54gig

2. why is this happening?

3. what do i do so it doesn't happen in the future?