cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What happens if I have both DLTs and normal tables in a single notebook?

hello_world
New Contributor III

I've just learned Delta Live Tables on Databricks Academy and have no environment to try it out.

I'm wondering what happens to the pipeline if the notebook consists of both normal tables and DLTs.

For example

  1. Table A
  2. DLT A that reads and cleans Table A
  3. Table B that enriches DLT A
  4. DLT B that does some aggregations on Table B

As you see, DLT B does not have a direct linkage to DLT A.

Will the pipeline still be able to track DLT A > Table B > DLT B? Or will this render an exception?

7 REPLIES 7

Aviral-Bhardwaj
Esteemed Contributor III

Thanks for sharing the doc, but all tables in the example are Delta Live Tables.

I wonder what happens to the pipeline if one of them is not a DLT.

Let's say the Bronze tables are DLTs, and the Silver table is NOT, but the golden tables are again DLTs. Will the pipeline still be able to track the Silver table?

Meghala
Valued Contributor II

@Aviral Bhardwaj​  thank you from my end

Meghala
Valued Contributor II

Thank you ​

Rishabh-Pandey
Honored Contributor II

hey ,@S L​  According to you , you have normal table table A and DLT table Table B , so it will give thrown an error that your upstream table is not streaming Live table and you need to create streaming live table Table a , if you want to use the output of that table to Table B .

then 3rd option , it will not throw any exception and will work as per my understanding , because your source can be a DLT table if you are using that table output to the normal table .

4th option will again give an error , because your source is not a DLT table and you are using the output of the source to the DLT table , which is not possible , i think

Rishabh Pandey

Thanks. This clarifies my confusion.

Does Table A have to be a STREAMING Live Table or it will work as long as it is a DLT? Just want to understand if streaming is also a pre-requisite.

Meghala
Valued Contributor II

even it was helped me as well thank you

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!