LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.

shan-databricks — Tue, 18 Feb 2025 13:27:53 GMT

Need help to resolve the issue

Error : com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [_LEGACY_ERROR_TEMP_DELTA_0007] A schema mismatch detected when writing to the Delta table.

I am using the below code and my JSON is dynamically changing daily basis

(spark.readStream

.format("cloudFiles")

.option("cloudFiles.format", "json")

.option("header", "true")

.option("cloudFiles.inferColumnTypes", "true")

.option("cloudFiles.schemaLocation", schema_location)

.load(input_path)

.writeStream.format("delta")

.option("checkpointLocation", checkpoint_location)

.trigger(availableNow=True)

.partitionBy(*partition_columns)

.option("mergeSchema", "true")

.option("badRecordsPath", f"{checkpoint_location}/badRecords")

.option("overwriteSchema", "true")

.option("schemaEvolutionMode", "addNewColumns")

.outputMode("append")

.start(output_path))

Re: LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.

cgrant — Thu, 27 Feb 2025 19:21:51 GMT

For datasets with constantly changing schemas, we recommend using the Variant type.

topic LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table. in Data Engineering

LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.

Re: LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.