topic Re: Urgent Assistance Needed – Unity Catalog Storage Access Failure & VM SKU Availability (Datab in Data Engineering

Urgent Assistance Needed – Unity Catalog Storage Access Failure & VM SKU Availability (Databricks on

suchitpathak08 — Thu, 27 Nov 2025 13:21:34 GMT

Hi everyone,

I’m running into two blocking issues while trying to run a Delta Live Tables (DLT) pipeline on Databricks (Azure). I’m hoping someone can help me understand what’s going wrong.

1. Unity Catalog cannot access underlying ADLS storage

Every DLT pipeline run fails with:

UNITY_CATALOG_INITIALIZATION_FAILED

INVALID_STATE.UC_CLOUD_STORAGE_ACCESS_FAILURE

AbfsRestOperationException

Even though:

The Access Connector’s managed identity has:

Storage Blob Data Owner
Storage Blob Data Contributor

My user account also has both of these roles.
ACLs on all containers (bronze, silver, gold, source, metastore, logs) have:
- The user
- The Access Connector managed identity

Read / Write / Execute for both:

All containers show correct ACLs (rwx), and IAM roles look correct at the storage account level.
But the pipeline still cannot initialize UC or access the storage.

2. VM size / SKU not available for DLT job compute

When the DLT pipeline tries to start a job cluster, I get:

The VM size you are specifying is not available (SkuNotAvailable)

QuotaExceeded: Required cores exceed available limit

Even small SKUs fail:

Standard_F4
Standard_DS3_v2
Standard_DS2_v2 (not visible in UI)
Standard_F2 (not visible in UI)

Azure CLI shows that many F-series SKUs exist in UK South, but in Databricks they fail to provision or don’t appear in the dropdown.

This makes it impossible to run even a minimal DLT cluster with 1 worker.

Additional Symptoms

The pipeline UI sometimes hides the “Advanced Options” section for compute configuration (no worker/driver selector).
Creating a manual cluster shows warnings like:

“This account may not have enough CPU cores to satisfy this request”
“Estimated available: 2, requested: 8”

Even when using a 4-core node with only one worker, the VM still fails with SkuNotAvailable.

What I’m trying to understand

Why UC still fails to access ADLS even when IAM + ACLs appear fully correct.
Whether this is a region-wide VM capacity issue in UK South.
Whether Databricks can enable smaller SKUs (F2 / DS2_v2) so DLT can run.
Whether this is a misconfiguration in my workspace or an Azure capacity limitation.
If the fix is:

an ACL/DL permissions change,
a quota request,
a VM selection change,
or migrating to another Azure region.

Any guidance would be hugely appreciated.

I’ve already:

Checked IAM
Checked container ACLs
Regenerated ACLs via Azure Portal
Validated the Access Connector identity
Tried multiple VM SKUs
Deleted and recreated the pipeline
Verified catalog and schema exist in Unity Catalog

Still getting the same two errors.

Thanks in advance to anyone who can help!

Re: Urgent Assistance Needed – Unity Catalog Storage Access Failure & VM SKU Availability (Datab

bianca_unifeye — Fri, 28 Nov 2025 12:11:03 GMT

Short Answer:

The UC error is almost always caused by the wrong identity being used in the Storage Credential / External Location, even if IAM + ACLs look correct.
The VM failures are typically quota + regional capacity issues in UK South, especially for older families like DSv2/F-series.
Fixes:
- Re-check the Storage Credential → External Location chain
- Increase quotas for a modern VM family
- Or try a newer region with better capacity

I will try to write step by step.

Re: Urgent Assistance Needed – Unity Catalog Storage Access Failure & VM SKU Availability (Datab

bianca_unifeye — Fri, 28 Nov 2025 12:13:25 GMT

UC Cloud Storage Access Failure (UC_CLOUD_STORAGE_ACCESS_FAILURE)

Even if IAM + ACLs look correct, Unity Catalog will not use ACLs directly. UC always accesses ADLS through a Storage Credential → External Location → Catalog chain.

A few things to verify:

Check which identity UC is actually using

Run:

SHOW EXTERNAL LOCATIONS;

DESCRIBE EXTERNAL LOCATION <your_location>;

The credential listed here must map to the Access Connector managed identity, not your user.

If the location shows a different storage credential, UC will try to access ADLS with the wrong identity → AbfsRestOperationException.

Confirm that the metastore root is UC-compatible

DESCRIBE CATALOG <your_catalog>;

The storage_location must itself be an external location with a valid credential.

If the metastore root was created before UC, it might not be correctly attached.

Check ADLS diagnostic logs

Enable "Blob Read/Write/Error" diagnostics on the storage account.

You will likely see failed requests from a principal different from the Access Connector MI — that’s the real smoking gun.

ACLs help, but are not sufficient

Unity Catalog requires:

Access Connector MI → StorageCredential assignment

StorageCredential → External Location

External Location → Catalog/Schema/Table

ACLs alone won’t fix UC initialisation.

Re: Urgent Assistance Needed – Unity Catalog Storage Access Failure & VM SKU Availability (Datab

bianca_unifeye — Fri, 28 Nov 2025 12:14:47 GMT

DLT pipelines always spin up job compute, and Azure is strict about SKU availability per region & per subscription.

Most common causes

Quota for that VM family is set to 2 vCPUs

Databricks shows:

“Estimated available: 2”

“QuotaExceeded”

The SKU exists in Azure CLI but Azure has no capacity for it in UK South

This is very common for older families like DS_v2 and F-series.

The Pipeline UI hides Advanced Options

This normally happens when Databricks can’t find any valid SKUs for job compute under your subscription constraints.

What to check

In Azure Portal → Subscription → Usage + quotas

Filter by:

Region: UK South

VM family: Dsv2, F-series, Dv3, Dv5, etc.

You will typically see vCPU limits like “2/2 used”.

Request a quota increase for at least:

Standard Dv3 Family vCPUs

Standard Dv5 Family vCPUs

(These have much better regional availability.)

Alternatively try a workspace in UK South 2 or North Europe, where clusters often provision successfully.