- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2025 06:35 AM
If you’re getting an error when ingesting into a Bronze table with DLT, make sure you’re using the DLT pattern end-to-end: (1) import DLT and declare the table with a decorator, e.g. import dlt then @Dlt.table(name="tbl_bronze_persons", comment="Raw persons data via Auto Loader"); (2) don’t call writeStream—in DLT the function must return a DataFrame, e.g. return (spark.readStream.format("cloudFiles").option("cloudFiles.format","json").option("cloudFiles.schemaLocation","/Volumes/<catalog>.<schema>.<vol>/schemas/persons").load("/mnt/<container>/landing/persons").withColumn("ingestion_timestamp", current_timestamp()).withColumn("source_file", input_file_name())); (3) run this notebook only from a DLT pipeline (not an interactive cluster), and set a UC target (catalog.schema) plus proper permissions; (4) avoid widget variables in DLT—use literal paths or pipeline parameters; (5) ensure the schemaLocation/checkpoint paths are in a reliable location (UC Volume or external storage) and that the workspace service principal/user has access. With the decorator + return pattern, proper Auto Loader options, and execution from a DLT pipeline, the Bronze ingestion should deploy cleanly.
Data Engineer | Machine Learning Engineer
LinkedIn: linkedin.com/in/wiliamrosa