I've set up Unity Catalog with an external location pointing to a storage account. For each schema, Iโve configured a dedicated container path. For example:
abfss://schemas@<storage_account>.dfs.core.windows.net/_unityStorage/schemas/<schema_id>
When I create a schema, a schema_id is generated. I expect this schema_id to be reflected as a folder under the schema container path, like:
/_unityStorage/schemas/<schema_id>
However, Iโve noticed that this folder doesnโt appear immediatelyโpresumably because no objects (like tables) exist yet.
Hereโs what Iโve observed:
- When I create a Delta table within the schema, I expect the table data to be stored under the schemaโs storage path.
- Similarly, when I create a DLT pipeline targeting the same schema, I expect the tables to be stored under the same schema path.
- But instead, a new schema ID folder gets created in the storage account under the schema containerโeven though the schema name is the same.
My question is: Under what conditions does Unity Catalog generate a new schema_id folder in the storage account, even when the schema name hasnโt changed?
Any insights or documentation references would be greatly appreciated!
