cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT with Unity Catalog pipeline not recognising tables from different schemas

Fz1
New Contributor III

I have 2 different schemas [silver and gold] under the same Unity Catalog.

We are trying to incrementally ingest data in both silver and gold layers.

The silver tables were created as streaming DLT tables using dlt.create_streaming_table(....) and the apply_merge(....) which is working perfectly

I have a DLT pipeline reading from the silver streaming DLT table and try to ingest to gold using the same concept:

dlt.create_streaming_table(GOLD_TABLE)

then 

apply_merge({UNITY_CATALOG_NAME}.{SCHEMA_SILVER}.{SOURCE_SILVER_TABLE} >>>> {GOLD_TABLE} . . . )

and the pipeline is throwing this error:

Failed to read dataset '{UNITY_CATALOG}.{SCHEMA_SILVER}.{TABLE_NAME}'. Dataset is not defined in the pipeline

 

Configuration:

  • The Catalog was created by specifying an ADL storage for it.
  • Schema was created without specifying any storage for it (because I faced another issue with the DLT pipeline, not allowing to execute pipeline for specified SCHEMA Location)
  • Schema details:
    • Catalog Type: MANAGED_CATALOG
    • Securable Type: SCHEMA
    • Securable Kind: SCHEMA_STANDARD
    • Browse Only: false
  • Catalog Details:
    • Storage root: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}
      Storage location: abfss://{CONTAINER}@{STORAGE_ACCOUNT}.dfs.core.windows.net/{FOLDER_NAME}/__unitystorage/catalogs/be07xxxxx-xxxxxxxxxxxx

PS: The same was working perfectly before using the Unity Catalog, by simply using the hive_metastore.

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group