LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.

[shan-databricks]

Need help to resolve the issue 

Error : com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [_LEGACY_ERROR_TEMP_DELTA_0007] A schema mismatch detected when writing to the Delta table.

I am using the below code and my JSON is dynamically changing daily basis

(spark.readStream

              .format("cloudFiles")

              .option("cloudFiles.format", "json")

              .option("header", "true")

              .option("cloudFiles.inferColumnTypes", "true")

              .option("cloudFiles.schemaLocation", schema_location)

              .load(input_path)

              .writeStream.format("delta")

              .option("checkpointLocation", checkpoint_location)

              .trigger(availableNow=True)

              .partitionBy(*partition_columns)

              .option("mergeSchema", "true")

              .option("badRecordsPath", f"{checkpoint_location}/badRecords")

              .option("overwriteSchema", "true")

              .option("schemaEvolutionMode", "addNewColumns")

              .outputMode("append")

              .start(output_path))

[cgrant]

For datasets with constantly changing schemas, we recommend using the Variant type.