cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?

J_M_W
Contributor

Hi there,

Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?

When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscription"

Essentially, I would like to have one notebook in the DLT pipeline that calls another notebook. The other "called" notebook creates a Delta Live Table.

For example...

The "caller" notebook, which will be attached to the DLT Pipeline, would look something like this...

dbutils.notebook.run('dlt_test_notebook', 3600)

And the notebook which will be "called" looks something like this...

import dlt
 
@dlt.table(name = 'test')
    
def f():
        
    # create dataframe
    df = spark.sql(query) 
 
    return df

Any help is much appreciated

1 ACCEPTED SOLUTION

Accepted Solutions

J_M_W
Contributor

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

View solution in original post

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @James Winfield​,

This article describes how to use Databricks notebooks to code complex workflows that use modular code, linked or embedded notebooks, and if-then-else logic.

Databricks has multiple pricing plans. If your current plan is no longer meeting all of your organization’s needs, you can upgrade to a higher-tier plan. The instructions for upgrading your account are different for pay-as-you-go and contract accounts.

Are you on Community Edition?

J_M_W
Contributor

Hi all.

@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.

After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - which is a shame 😞

Hope this helps someone else who is struggling with this!

Thanks, JMW

Does that mean you can run dbutils command?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group