I have 2 different schemas [silver and gold] under the same Unity Catalog.
We are trying to incrementally ingest data in both silver and gold layers.
The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the apply_merge(....) which is working perfectly
I have a DLT pipeline reading from the silver streaming DLT table and try to ingest to gold using the same concept:
dlt.create_streaming_table(GOLD_TABLE)
then
apply_merge({UNITY_CATALOG_NAME}.{SCHEMA_SILVER}.{SOURCE_SILVER_TABLE} >>>> {GOLD_TABLE} . . . )
and the pipeline is throwing this error:
Failed to read dataset '{UNITY_CATALOG}.{SCHEMA_SILVER}.{TABLE_NAME}'. Dataset is not defined in the pipeline
Configuration:
- The Catalog was created by specifying an ADL storage for it.
- Schema was created without specifying any storage for it (because I faced another issue with the DLT pipeline, not allowing to execute pipeline for specified SCHEMA Location)
- Schema details:
- Catalog Type: MANAGED_CATALOG
- Securable Type: SCHEMA
- Securable Kind: SCHEMA_STANDARD
- Browse Only: false
- Catalog Details:
- Storage root: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}
Storage location: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}/__unitystorage/catalogs/be07xxxxx-xxxxxxxxxxxx
PS: The same was working perfectly before using the Unity Catalog, by simply using the hive_metastore.
