DLT + Unity Catalogue Issue accessing Dataset not defined in the pipeline

Fz1 — Mon, 03 Jul 2023 15:28:25 GMT

I have 2 different schemas [silver and gold] under the same Unity Catalog.

We are trying to incrementally ingest data in both silver and gold layers.

The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the apply_merge(....) which is working perfectly

I have a DLT pipeline reading from the silver streaming DLT table and try to ingest to gold using the same concept:

dlt.create_streaming_table(GOLD_TABLE)

then

apply_merge({UNITY_CATALOG_NAME}.{SCHEMA_SILVER}.{SOURCE_SILVER_TABLE} >>>> {GOLD_TABLE} . . . )

and the pipeline is throwing this error:

Failed to read dataset '{UNITY_CATALOG}.{SCHEMA_SILVER}.{TABLE_NAME}'. Dataset is not defined in the pipeline

Configuration:

The Catalog was created by specifying an ADL storage for it.
Schema was created without specifying any storage for it (because I faced another issue with the DLT pipeline, not allowing to execute pipeline for specified SCHEMA Location)
Schema details:
- Catalog Type: MANAGED_CATALOG
- Securable Type: SCHEMA
- Securable Kind: SCHEMA_STANDARD
- Browse Only: false
Catalog Details:
- Storage root: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}
  Storage location: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}/__unitystorage/catalogs/be07xxxxx-xxxxxxxxxxxx

PS: The same was working perfectly before using the Unity Catalog, by simply using the hive_metastore.

topic DLT + Unity Catalogue Issue accessing Dataset not defined in the pipeline in Data Engineering