Massive increase in the number of "GetBlobProperties" operations

[syazwansuhaimi]

I had a massive increase in the volume of "GetBlobProperties" operations in my Azure Blob Storage account. The storage logs indicate that all the extra operations have IPs attributed to my Databricks resource group. I haven't made any changes to my resource configurations, so I'm confused on what could have led to this increase. I appreciate any input on this issue.

[Vidhi_Khaitan]

Massive increase in "GetBlobProperties" operations in your Azure Blob Storage account could be due to the following

1. Delta Tables and _delta_log Metadata Access
If you're using Delta Lake, Databricks reads blob properties (e.g., last-modified time, size) from the _delta_log directory to check for updates. A new notebook, workflow, or user action that frequently reads or lists a Delta table can lead to repeated GetBlobProperties operations on the json commit files and checkpoints in that directory.

Check if:
New notebooks or dashboards recently started querying Delta tables more often.
Auto-refresh features in notebooks or dashboards are enabled.
VACUUM, DESCRIBE HISTORY, or schema evolution commands are being run more frequently.

2. Databricks File System (DBFS) Listing
Any dbutils.fs.ls() or similar calls made recursively over folders in DBFS backed by Azure Blob Storage can trigger many GetBlobProperties calls. These are commonly used in diagnostics, logging, or monitoring code.

3. Unity Catalog or External Table Audits
If you're using Unity Catalog or have external Hive Metastore tables that point to blob-backed data, background jobs or access control audits can trigger metadata reads.

Widgets or jobs that query tables repeatedly, especially in a multi-user workspace, can inadvertently lead to a flood of GetBlobProperties.

I hope this gives you some clarity!