cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

J_M_W
Contributor

Hi there,

Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?

When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscription"

Essentially, I would like to have one notebook in the DLT pipeline that calls another notebook. The other "called" notebook creates a Delta Live Table.

For example...

The "caller" notebook, which will be attached to the DLT Pipeline, would look something like this...

dbutils.notebook.run('dlt_test_notebook', 3600)

And the notebook which will be "called" looks something like this...

import dlt
 
@dlt.table(name = 'test')
    
def f():
        
    # create dataframe
    df = spark.sql(query) 
 
    return df

Any help is much appreciated

1 ACCEPTED SOLUTION

Accepted Solutions

J_M_W
Contributor

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

View solution in original post

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @James Winfield​,

This article describes how to use Databricks notebooks to code complex workflows that use modular code, linked or embedded notebooks, and if-then-else logic.

Databricks has multiple pricing plans. If your current plan is no longer meeting all of your organization’s needs, you can upgrade to a higher-tier plan. The instructions for upgrading your account are different for pay-as-you-go and contract accounts.

Are you on Community Edition?

J_M_W
Contributor

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

Does that mean you can run dbutils command?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!