Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2025 07:12 AM
Hi @a_user12 DLT is designed to automatically evolve the schema of tables as your pipeline logic changes. If your code returns a DataFrame with new columns, DLT will add those columns to the table automatically. There is no built-in option to prevent this or to enforce a fixed schema.
1) you can try enforcing explicit schema definitions in your pipeline code??
also try , Disablng schema auto merge globally
2) spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", "false")
example :
1)
# Define schema for payload
payload_schema = StructType([
StructField("field1", StringType(), True),
StructField("field2", IntegerType(), True)
])
.withColumn("payload", from_json(col("payload"), payload_schema))
