Delta Live Table Pipeline with Multiple Notebooks

[Dave_Nithio]

I have two notebooks created for my Delta Live Table pipeline. The first is a utils notebook with functions I will be reusing for other pipelines. The second contains my actual creation of the delta live tables. I added both notebooks to the pipeline settings

But the pipeline fails with the error 'Failed to execute python command for notebook' pointing to the function I created in my utils notebook. Alternatively, I attempted to use a %run magic command to force the utils notebook to run first, but it did not work. I was given the warning that magic commands are not supported. Is there any way to force the Delta Live Table Pipeline to load my utils notebook first so that its functions can be referenced while building the pipeline?

[Dave_Nithio]

Per another question we are unable to use either magic commands or dbutils.notebook.run with the pro level databricks account or Delta Live Tables. Are there any other solutions for utilizing generic functions from other notebooks within a Delta Live Table pipeline?

[Vivian_Wilfred · Accepted solution]

Hi @Dave Wilson​ %run or dbutils is not supported in DLT. This is intentionally disabled because DLT is declarative and we cannot perform data movement on our own.

To answer your first query, there is unfortunately no option to make the utils notebook run first. The only option is to combine utils and your main notebooks together. This does not address the reusability aspect in DLT and we have raised this feature request with the product team. The engineering team is working internally to address this issue. We can soon expect a feature that would address this usecase.

Thanks.

[fecavalc08]

Hi @Vivian Wilfred​ and @Dave Wilson​ we solved our reusability code with repos and pointing the code to our main code:

sys.path.append(os.path.abspath('/Workspace/Repos/[your repo]/[folder with the python scripts'))

from your_class import *

It just works if your reusable code is in python. Also depending on what you want to do we noticed that DLT is always executed as the last piece of the code no matter what is the position in the script

[ssudhakarpatil]

Could you help me with explain this in detail?

Lets i have notebook abc which is reusable and pqr is the one i will be mentioning in dlt pipeline.

how do i call functions from abc pipeline in pqr?