Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

[J_M_W]

Hi there,

Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?

When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscription"

Essentially, I would like to have one notebook in the DLT pipeline that calls another notebook. The other "called" notebook creates a Delta Live Table.

For example...

The "caller" notebook, which will be attached to the DLT Pipeline, would look something like this...

dbutils.notebook.run('dlt_test_notebook', 3600)

And the notebook which will be "called" looks something like this...

import dlt
 
@dlt.table(name = 'test')
    
def f():
        
    # create dataframe
    df = spark.sql(query) 
 
    return df

Any help is much appreciated

[J_M_W · Accepted solution]

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

[ssudhakarpatil]

Does that mean you can run dbutils command?