I set up an Azure Data Lake Storage (ADLS) account with containers named metastore, bronze, silver, gold, and source. I created a Unity Catalog metastore in Databricks via the admin console, and I created a container called metastore in my Data Lake. I defined external locations for each container (e.g., abfss://bronze@<storage_account>.dfs.core.windows.net/) and created a catalog without specifying a location, assuming it would use the metastore's default location. I also created schemas (bronze, silver, gold) and assigned each schema to the corresponding container's external location (e.g., bronze schema mapped to the bronze container).
In my source container, I have a folder structure: customers/customers.csv.
I built a Delta Live Tables (DLT) pipeline with the following configuration:
-- Bronze table
CREATE OR REFRESH STREAMING TABLE my_catalog.bronze.customers
AS
SELECT *, current_timestamp() AS ingest_ts, _metadata.file_name AS source_file
FROM STREAM read_files(
'abfss://source@<storage_account>.dfs.core.windows.net/customers',
format => 'csv'
);
-- Silver table
CREATE OR REFRESH STREAMING TABLE my_catalog.silver.customers
AS
SELECT *, current_timestamp() AS process_ts
FROM STREAM my_catalog.bronze.customers
WHERE email IS NOT NULL;
-- Gold materialized view
CREATE OR REFRESH MATERIALIZED VIEW my_catalog.gold.customers
AS
SELECT count(*) AS total_customers
FROM my_catalog.silver.customers
GROUP BY country;
Why are my tables stored under this unity/schemas/<schema_id>/tables/<table_id> structure instead of directly in customers/parquet_files with a _delta_log folder in the respective containers?
How can I configure my DLT pipeline or Unity Catalog setup to ensure the tables are stored in the bronze, silver, and gold containers with a folder structure like customers/parquet_files and _delta_log?
In industry-level projects, how do teams typically manage table storage locations and folder structures in ADLS when using Unity Catalog and Delta Live Tables? Are there best practices or common configurations to ensure a clean, predictable folder structure for bronze, silver, and gold layers?
