Did something change with the setup of Unity Catalogs?

[Jefke]

Just some background. I deliver quite some trainings on Azure Databricks and in these trainings students have to provisions their own Azure Databricks workspace and UC. In the past this always went smoothly.

Go to the Azure Portal 

1. Create a storage account
2. Create workspace
3. Grant Storage Blob Contributor on the storage account to an Access Connector of Azure Databricks
4. Create a new UC metastore and link the workspace to it.
5. Done.

This whole procedure did actually create the external location and connection objects automatically.

When we now try to repeat these same 5 steps we get an error creating the UC metastore:

parent external location for path `abfss://<>@<>.dfs.core.windows.net/` does not exist.

Some other strange thing is that the abfss:// never had to be added in the past. Now you must manually add it. In the docs it is still saying "The abfss:// prefix is added automatically."

To get the creation of the of the UC metastore to succeed, the Credential and External Location must be created upfront. Once that is done, the above error message does not appear anymore...

Any idea what is causing the changed behaviour?

[ogbenisho · Accepted solution]

The changed behavior is due to updates in how Unity Catalog (UC) handles external locations and storage credentials in Azure Databricks. Previously, UC automatically created the external location and credential objects when provisioning a metastore, but now these objects must be explicitly defined before metastore creation.

What Changed

Earlier, creating a UC metastore would implicitly generate the required storage credential and external location. This is no longer the case. You now need to manually create:

• A storage credential (linked to your Access Connector or managed identity).
• An external location (with the abfss:// path explicitly defined).

The error parent external location for path abfss://<>@<>.dfs.core.windows.net/ does not exist occurs because UC now validates that the external location object exists before linking the metastore.

Microsoft’s documentation still states that the abfss:// prefix is added automatically, but in practice Databricks Runtime 11.3+ and UC updates changed this behavior. The change aligns with UC’s stricter governance model: external locations are now treated as securable objects that must be explicitly created and managed.

You can adopt this as your current correct procedure

1. Create a storage account in Azure.
2. Create workspace in Azure Databricks.
3. Grant Storage Blob Contributor role on the storage account to the Access Connector.
4. Create storage credential in UC (linked to the Access Connector).
5. Create external location in UC with the explicit abfss:// path.
6. Create UC metastore and link the workspace to it.

Update your training scripts to include manual creation of storage credential and external location before metastore creation. Explain to students that this is part of UC’s stricter governance model. Highlight that older documentation may mislead them into skipping these steps.

[Jefke]

Hi Ogbenisho,

I have one follow up question on the 6 steps you describe:

In step 4 and 5 you assume there is already a UC metastore to create the credentials and external location in.
And then in step 6 you create it the UC metastore

Bu what if you have a workspace that does not have a UC metastore yet?

I always thought you needed a UC metastore to create these 2 artifacts?

Thanks again!

[ogbenisho · Accepted solution]

You are very right. Before, you won’t be able to create storage credentials or external locations until a UC metastore existed, because those objects were scoped inside the metastore. That’s why everything used to work fine once the metastore was provisioned.

Now, they’re no longer scoped inside a metastore, but instead exist at the account level. Meaning you must create them before the metastore, so that when you provision the metastore, it can reference those pre-existing objects. The error you saw (parent external location … does not exist) is UC enforcing that the metastore must point to a valid external location object.

Here is the revised corrected sequence:

1. Create storage account in Azure.
2. Create workspace in Azure Databricks.
3. Grant Storage Blob Contributor role on the storage account to the Access Connector.
4. Create storage credential (account-level UC object, linked to the Access Connector).
5. Create external location (account-level UC object, with explicit abfss:// path).
6. Create UC metastore and link the workspace to it, referencing the external location.

So, steps 4 and 5 no longer depend on the metastore being present, as they live at the account level. That’s the key shift.