LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch det...

shan-databricks · ‎02-18-2025

Need help to resolve the issue

Error : com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [_LEGACY_ERROR_TEMP_DELTA_0007] A schema mismatch detected when writing to the Delta table.

I am using the below code and my JSON is dynamically changing daily basis

(spark.readStream

.format("cloudFiles")

.option("cloudFiles.format", "json")

.option("header", "true")

.option("cloudFiles.inferColumnTypes", "true")

.option("cloudFiles.schemaLocation", schema_location)

.load(input_path)

.writeStream.format("delta")

.option("checkpointLocation", checkpoint_location)

.trigger(availableNow=True)

.partitionBy(*partition_columns)

.option("mergeSchema", "true")

.option("badRecordsPath", f"{checkpoint_location}/badRecords")

.option("overwriteSchema", "true")

.option("schemaEvolutionMode", "addNewColumns")

.outputMode("append")

.start(output_path))

cgrant · ‎02-27-2025

For datasets with constantly changing schemas, we recommend using the Variant type.

LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.