cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live Table run outside out pipeline

ruoyuqian
New Contributor II

I have created a notebook for my Delta Live Table pipeline and it runs without errors however if I run the notebook alone in my cluster it,  says not allowed and show this error.  Does it mean I can only run delta live table in the pipeline and cannot run locally on a cluster? because if I run it via DLT it will cost me extra money to do so. 

import dlt
from pyspark.sql.functions import *
from pyspark.sql.functions import col, trim, split, concat, current_timestamp, row_number, expr, hash as pyspark_hash
from pyspark.sql.window import Window

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Rishabh-Pandey
Esteemed Contributor

Hi @ruoyuqian Delta Live Tables (DLT) have specific execution contexts and dependencies that are managed within their pipeline environment. This is why the code runs successfully only when executed within the pipeline, as DLT creates its own job cluster tailored to its dependencies rather than relying on a dedicated cluster. If you're concerned about costs, I recommend reviewing Databricks' compute pricing and exploring optimization strategies to reduce expenses effectively

Rishabh Pandey

View solution in original post

1 REPLY 1

Rishabh-Pandey
Esteemed Contributor

Hi @ruoyuqian Delta Live Tables (DLT) have specific execution contexts and dependencies that are managed within their pipeline environment. This is why the code runs successfully only when executed within the pipeline, as DLT creates its own job cluster tailored to its dependencies rather than relying on a dedicated cluster. If you're concerned about costs, I recommend reviewing Databricks' compute pricing and exploring optimization strategies to reduce expenses effectively

Rishabh Pandey

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group