Datatype changed while writing in delta format

[AnkithP]

Hello team,

I'm encountering an issue with my batch processing job. Initially, I write the job in overwrite mode with overwrite schema set to true. However, when I attempt to write the next batch in append mode, it fails due to a change in the datatype of one column. How can I handle this situation when I don't know the datatypes and don't have an external schema to refer to?

[Kaniz]

Hi @AnkithP, 

• When reading data in overwrite mode, use schema inference to automatically detect the column datatypes. Most batch processing frameworks (e.g., Apache Spark™, Apache Flink) provide this feature.
• Store the inferred schema (e.g., as a JSON file) for future reference.