K_Anudeep
Databricks Employee
Databricks Employee

Hello @Dhruv-22 ,

No—mergeSchema doesn’t auto-widen an incoming INT column to a table’s BIGINT (nor does it auto-cast). mergeSchema mainly helps add new columns (and historically only a tiny set of numeric upcasts), but it won’t change an existing column’s type or cast your DataFrame to match the table. That’s why you see [DELTA_FAILED_TO_MERGE_FIELDS]

Doc Link: https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html

However, you can rewrite the schema and data using 

df_with_new_types
  .write.format("delta")
  .mode("overwrite")
  .option("overwriteSchema", "true")
  .saveAsTable(table_name))

If you are using DBR 15.4 and above, you can try enabling type_widening

Doc Link : https://docs.databricks.com/aws/en/delta/type-widening#enable-type-widening

Anudeep

View solution in original post