topic Re: What happens if I have both DLTs and normal tables in a single notebook? in Data Engineering

What happens if I have both DLTs and normal tables in a single notebook?

hello_world — Sun, 25 Dec 2022 02:48:37 GMT

I've just learned Delta Live Tables on Databricks Academy and have no environment to try it out.

I'm wondering what happens to the pipeline if the notebook consists of both normal tables and DLTs.

For example

Table A
DLT A that reads and cleans Table A
Table B that enriches DLT A
DLT B that does some aggregations on Table B

As you see, DLT B does not have a direct linkage to DLT A.

Will the pipeline still be able to track DLT A > Table B > DLT B? Or will this render an exception?

Re: What happens if I have both DLTs and normal tables in a single notebook?

Aviral-Bhardwaj — Sun, 25 Dec 2022 03:05:58 GMT

This doc will help you https://www.databricks.com/discover/pages/getting-started-with-delta-live-tables#prerequisites

Re: What happens if I have both DLTs and normal tables in a single notebook?

hello_world — Sun, 25 Dec 2022 05:58:02 GMT

Thanks for sharing the doc, but all tables in the example are Delta Live Tables.

I wonder what happens to the pipeline if one of them is not a DLT.

Let's say the Bronze tables are DLTs, and the Silver table is NOT, but the golden tables are again DLTs. Will the pipeline still be able to track the Silver table?

Re: What happens if I have both DLTs and normal tables in a single notebook?

Rishabh-Pandey — Mon, 26 Dec 2022 07:15:14 GMT

hey ,@S L According to you , you have normal table table A and DLT table Table B , so it will give thrown an error that your upstream table is not streaming Live table and you need to create streaming live table Table a , if you want to use the output of that table to Table B .

then 3rd option , it will not throw any exception and will work as per my understanding , because your source can be a DLT table if you are using that table output to the normal table .

4th option will again give an error , because your source is not a DLT table and you are using the output of the source to the DLT table , which is not possible , i think

Re: What happens if I have both DLTs and normal tables in a single notebook?

hello_world — Mon, 26 Dec 2022 09:28:23 GMT

Thanks. This clarifies my confusion.

Does Table A have to be a STREAMING Live Table or it will work as long as it is a DLT? Just want to understand if streaming is also a pre-requisite.

Re: What happens if I have both DLTs and normal tables in a single notebook?

Meghala — Mon, 26 Dec 2022 10:46:45 GMT

even it was helped me as well thank you

Re: What happens if I have both DLTs and normal tables in a single notebook?

Meghala — Mon, 26 Dec 2022 10:47:51 GMT

@Aviral Bhardwaj thank you from my end

Re: What happens if I have both DLTs and normal tables in a single notebook?

Meghala — Mon, 26 Dec 2022 14:33:13 GMT

Thank you