topic Re: DLT pipeline in Data Engineering

DLT pipeline

Rishitha — Fri, 22 Sep 2023 18:28:09 GMT

Hi all!

I have a question about setting a target schema. How to set different targets for 2 different tables in the same delta live table pipeline.

We have 2 target schemas in a database Bronze_chema and silver_schema.

The pipeline has a streaming raw table (Bronze table) Table A and a processed table (Silver Table) derived from the bronze table, Table B. When I set the target schema in the Workflow settings it defaults both the tables to that target schema.

Table A -> Bronze_schema

TableB -> Bronze_schema

But I want

Table A -> Bronze_schema and

Table B -> Silver_schema, as part of the same delta live table pipeline. please suggest how to set the targets as part of pipeline code

Re: DLT pipeline

Rishitha — Fri, 22 Sep 2023 18:34:34 GMT

I have also asked the question in the office hours, they said it will be a new feature, announced in the summit too, but there is a workaround to do it now. Anyone implemented the workaround? '

Re: DLT pipeline

btafur — Fri, 22 Sep 2023 18:45:40 GMT

At the moment, with the current features available, it is not possible to have multiple schemas in one DLT pipeline. In some cases, it is possible to achieve something similar by creating multiple DLT pipelines for the different schemas.

Re: DLT pipeline

Rishitha — Fri, 22 Sep 2023 19:44:16 GMT

Thanks again @btafur

Hoping for this feature to release soon!