topic Re: Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline? in Data Engineering

Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

J_M_W — Thu, 06 Oct 2022 18:37:11 GMT

Hi there,

Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?

When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscription"

Essentially, I would like to have one notebook in the DLT pipeline that calls another notebook. The other "called" notebook creates a Delta Live Table.

For example...

The "caller" notebook, which will be attached to the DLT Pipeline, would look something like this...

dbutils.notebook.run('dlt_test_notebook', 3600)

And the notebook which will be "called" looks something like this...

import dlt
 
@dlt.table(name = 'test')
    
def f():
        
    # create dataframe
    df = spark.sql(query) 
 
    return df

Any help is much appreciated

Re: Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

J_M_W — Mon, 10 Oct 2022 08:31:43 GMT

Hi all.

@Kaniz Fatma thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

Re: Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

ssudhakarpatil — Tue, 02 May 2023 14:06:00 GMT

Does that mean you can run dbutils command?