DLT Pipeline on Hive Metastore

[Sreyasi_Thakur]

I am creating a DLT pipeline on Hive Metastore (destination is Hive Metastore) and using a notebook within the pipeline which reads a unity catalog table. But, I am getting an error- [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster.

Is it a limitation of DLT that if we define the pipeline destination as Hive then it won’t allow to read tables from Unity Catalog?

[Kaniz_Fatma]

Hi @Sreyasi_Thakur, Yes, this is a known limitation. When you define the pipeline destination as Hive Metastore, you cannot read tables from Unity Catalog within the same pipeline. Delta Live Tables (DLT) pipelines can either use the Hive Metastore or Unity Catalog, but not both s...

1. If possible, switch your pipeline to use Unity Catalog for both reading and writing tables.
2. Create separate pipelines where one pipeline reads from Unity Catalog and writes to an intermediate storage (like Delta Lake), and another pipeline reads from this intermediate storage and writes to Hive Metastore.

Would you like more details on how to implement either of these solutions?

[Sreyasi_Thakur]

Hi @Kaniz_Fatma - yes, please share more details on the 2nd solution.