topic Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType in Data Engineering

Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType

vijaykumarbotla — Fri, 26 May 2023 08:12:42 GMT

I am have imported a csv file using spark.read method, i have used custom schema and declared the type of the column as string.

i have delta table and the type of the column in the table is also string.

I am getting failed to merge fields errors in spark.write.format('delta') command,

can you please suggest.

Re: Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType

-werners- — Fri, 26 May 2023 09:25:06 GMT

apparently the schema was not applied to your df.

You will have to check why (assuming the delta table col is indeed of type string). Without sharing code it is impossible to know.

Re: Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType

pvignesh92 — Fri, 26 May 2023 10:21:19 GMT

@Vijay Kumar

Check your schema of the table using DESCRIBE TABLE and the dataframe you loaded using df.printSchema() and compare which column has a mismatch. You need to see if your custom schema is matching the table schema.
MergeSchema will support only the schemas that it can typecast.
If you want to completely recreate the table schema with the ones of dataframes, you can use overwriteSchema option.

Please share your code so that we can look and help you.

Re: Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType

Anonymous — Sat, 27 May 2023 08:07:28 GMT

Hi @Vijay Kumar

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType

vijaykumarbotla — Mon, 29 May 2023 16:12:17 GMT

Hi All,

the issue is resolved, i have executed column conversion and from next run the code is working fine.

df = spark.read.format("delta").load("/mnt/dev/deltav2/X")

df= df.withColumn("LIFNR", df.LIFNR.cast("string"))

df.write.format('delta').option("overwriteSchema", "true").mode('overwrite').save("/mnt/dev/deltav2/X")