topic Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster in Data Engineering

Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster

ManojkMohan — Thu, 14 Aug 2025 20:42:23 GMT

Use Case Description:

Manually uploading orders data into databricks

Then moving it into a bronze layer using the below code

technical Code used python

Getting error:

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

Trying to debug myself, appreciate some help in parallel

Re: Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster

WiliamRosa — Fri, 15 Aug 2025 13:35:22 GMT

If you’re getting an error when ingesting into a Bronze table with DLT, make sure you’re using the DLT pattern end-to-end: (1) import DLT and declare the table with a decorator, e.g. import dlt then @Dlt.table(name="tbl_bronze_persons", comment="Raw persons data via Auto Loader"); (2) don’t call writeStream—in DLT the function must return a DataFrame, e.g. return (spark.readStream.format("cloudFiles").option("cloudFiles.format","json").option("cloudFiles.schemaLocation","/Volumes/<catalog>.<schema>.<vol>/schemas/persons").load("/mnt/<container>/landing/persons").withColumn("ingestion_timestamp", current_timestamp()).withColumn("source_file", input_file_name())); (3) run this notebook only from a DLT pipeline (not an interactive cluster), and set a UC target (catalog.schema) plus proper permissions; (4) avoid widget variables in DLT—use literal paths or pipeline parameters; (5) ensure the schemaLocation/checkpoint paths are in a reliable location (UC Volume or external storage) and that the workspace service principal/user has access. With the decorator + return pattern, proper Auto Loader options, and execution from a DLT pipeline, the Bronze ingestion should deploy cleanly.

Re: Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster

ilir_nuredini — Fri, 15 Aug 2025 13:40:39 GMT

Hello @ManojkMohan ,

If you try to run a DLT pipeline on an (e.g.) all-purpose compute cluster, it will fail. DLT pipelines require a DLT job computer cluster.

To run it, create a new pipeline in the Databricks UI, assign your DLT notebook to it, and start the pipeline:

The reason it does not work on all-purpose compute is that the DLT module is not supported on Spark Connect clusters. The DLT runtime runs on a specialized job compute environment, which is different from the standard all-purpose compute runtime. It provides capabilities and configurations tailored specifically for Delta Live Tables. Because of these differences, a DLT pipeline cannot be executed on any all-purpose cluster, it must run on a DLT job cluster.

Hope that helps. Let me know if you have any other questions!

Best, Ilir

Re: Ingest into bronze Table: getting error Delta Live Tables (DLT) is not supported on this cluster

ManojkMohan — Tue, 19 Aug 2025 18:02:38 GMT

Issue Encountered:
Received the following error when attempting to run Delta Live Tables (DLT) code:

"The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code."

Short-Term Workaround (Bronze Table):
To proceed without DLT, used standard Delta Lake APIs:

raw_df.write.format("delta").mode("overwrite").saveAsTable("bronze.ipl_raw") df = spark.read.table("bronze.ipl_raw")

Now trying a prototype with DLT pipeline