Delta Live Table run outside out pipeline

ruoyuqian — Wed, 07 Aug 2024 05:12:18 GMT

I have created a notebook for my Delta Live Table pipeline and it runs without errors however if I run the notebook alone in my cluster it, says not allowed and show this error. Does it mean I can only run delta live table in the pipeline and cannot run locally on a cluster? because if I run it via DLT it will cost me extra money to do so.

import dlt from pyspark.sql.functions import * from pyspark.sql.functions import col, trim, split, concat, current_timestamp, row_number, expr, hash as pyspark_hash from pyspark.sql.window import Window The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

Re: Delta Live Table run outside out pipeline

Rishabh-Pandey — Wed, 07 Aug 2024 05:55:43 GMT

Hi @ruoyuqian Delta Live Tables (DLT) have specific execution contexts and dependencies that are managed within their pipeline environment. This is why the code runs successfully only when executed within the pipeline, as DLT creates its own job cluster tailored to its dependencies rather than relying on a dedicated cluster. If you're concerned about costs, I recommend reviewing Databricks' compute pricing and exploring optimization strategies to reduce expenses effectively

topic Re: Delta Live Table run outside out pipeline in Data Engineering

Delta Live Table run outside out pipeline

Re: Delta Live Table run outside out pipeline