Delta Live Table run outside out pipeline

[ruoyuqian]

I have created a notebook for my Delta Live Table pipeline and it runs without errors however if I run the notebook alone in my cluster it,  says not allowed and show this error.  Does it mean I can only run delta live table in the pipeline and cannot run locally on a cluster? because if I run it via DLT it will cost me extra money to do so. 

import dlt
from pyspark.sql.functions import *
from pyspark.sql.functions import col, trim, split, concat, current_timestamp, row_number, expr, hash as pyspark_hash
from pyspark.sql.window import Window

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

[Rishabh-Pandey · Accepted solution]

Hi @ruoyuqian Delta Live Tables (DLT) have specific execution contexts and dependencies that are managed within their pipeline environment. This is why the code runs successfully only when executed within the pipeline, as DLT creates its own job cluster tailored to its dependencies rather than relying on a dedicated cluster. If you're concerned about costs, I recommend reviewing Databricks' compute pricing and exploring optimization strategies to reduce expenses effectively