Does anyone know if there's a way to specify an alternate Unity schema in a DLT workflow using the @Dlt.table syntax? In my case, Iām looping through folders in Azure datalake storage to ingest data. Iād like those folders to get created in different schemas within a single (or multiple) unity catalog.
For example:
- Catalog1 : schemaA : tableA
- Catalog1 : schemaB : tableB
- Catalog2 : schemaC : tableC
I tried specifying the 3 dot nomenclature in the table name in my @Dlt.table syntax and received an error āINVALID_PARAMETER_VALUE.INVALID_FIELD CreateStagingTable name is not a valid name.ā Specifying the āpathā attribute also returns an error that I canāt do that with Unity. When creating the workflow Iām allowed to leave the schema blank but I canāt figure out how to specify it in my code.
Iāve been googling around for a couple days and couldnāt find an answer. Would love if you could suggest a solution that didnāt involve having to create separate workflows for each schema.