Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster

[ManojkMohan]

Use Case Description:

Manually uploading orders data into databricks 

Then moving it into a bronze layer using the below code

technical Code used python

Getting error:

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

Trying to debug myself, appreciate some help in parallel

[WiliamRosa]

If you’re getting an error when ingesting into a Bronze table with DLT, make sure you’re using the DLT pattern end-to-end: (1) import DLT and declare the table with a decorator, e.g. import dlt then @Dlt.table(name="tbl_bronze_persons", comment="Raw persons data via Auto Loader"); (2) don’t call writeStream—in DLT the function must return a DataFrame, e.g. return (spark.readStream.format("cloudFiles").option("cloudFiles.format","json").option("cloudFiles.schemaLocation","/Volumes/<catalog>.<schema>.<vol>/schemas/persons").load("/mnt/<container>/landing/persons").withColumn("ingestion_timestamp", current_timestamp()).withColumn("source_file", input_file_name())); (3) run this notebook only from a DLT pipeline (not an interactive cluster), and set a UC target (catalog.schema) plus proper permissions; (4) avoid widget variables in DLT—use literal paths or pipeline parameters; (5) ensure the schemaLocation/checkpoint paths are in a reliable location (UC Volume or external storage) and that the workspace service principal/user has access. With the decorator + return pattern, proper Auto Loader options, and execution from a DLT pipeline, the Bronze ingestion should deploy cleanly.

[ManojkMohan]

Issue Encountered:
Received the following error when attempting to run Delta Live Tables (DLT) code:

"The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code."

Short-Term Workaround (Bronze Table):
To proceed without DLT, used standard Delta Lake APIs:

raw_df.write.format("delta").mode("overwrite").saveAsTable("bronze.ipl_raw") df = spark.read.table("bronze.ipl_raw")

Now trying a prototype with DLT pipeline

[ilir_nuredini]

Hello @ManojkMohan ,

If you try to run a DLT pipeline on an (e.g.) all-purpose compute cluster, it will fail. DLT pipelines require a DLT job computer cluster.

To run it, create a new pipeline in the Databricks UI, assign your DLT notebook to it, and start the pipeline:

The reason it does not work on all-purpose compute is that the DLT module is not supported on Spark Connect clusters. The DLT runtime runs on a specialized job compute environment, which is different from the standard all-purpose compute runtime. It provides capabilities and configurations tailored specifically for Delta Live Tables. Because of these differences, a DLT pipeline cannot be executed on any all-purpose cluster, it must run on a DLT job cluster.

Hope that helps. Let me know if you have any other questions!

Best, Ilir