Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

[J_M_W]

Hi there,

Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?

When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscription"

Essentially, I would like to have one notebook in the DLT pipeline that calls another notebook. The other "called" notebook creates a Delta Live Table.

For example...

The "caller" notebook, which will be attached to the DLT Pipeline, would look something like this...

dbutils.notebook.run('dlt_test_notebook', 3600)

And the notebook which will be "called" looks something like this...

import dlt
 
@dlt.table(name = 'test')
    
def f():
        
    # create dataframe
    df = spark.sql(query) 
 
    return df

Any help is much appreciated

[Kaniz_Fatma]

Hi @James Winfield​,

This article describes how to use Databricks notebooks to code complex workflows that use modular code, linked or embedded notebooks, and if-then-else logic.

Databricks has multiple pricing plans. If your current plan is no longer meeting all of your organization’s needs, you can upgrade to a higher-tier plan. The instructions for upgrading your account are different for pay-as-you-go and contract accounts.

• If you have a pay-as-you-go account, follow these instructions to upgrade to another plan.
• If you have a contract with Databricks, contact your Databricks representative to manage your subscription level.
• To upgrade from Databricks Community Edition to the complete Databricks platform, sign up for the Databricks Free Trial or contact a Databricks representative. For more information, see Sign up for a free trial.

Are you on Community Edition?

[J_M_W · Accepted solution]

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

[ssudhakarpatil]

Does that mean you can run dbutils command?