05-26-2023 01:12 AM
I am have imported a csv file using spark.read method, i have used custom schema and declared the type of the column as string.
i have delta table and the type of the column in the table is also string.
I am getting failed to merge fields errors in spark.write.format('delta') command,
can you please suggest.
05-29-2023 09:12 AM
Hi All,
the issue is resolved, i have executed column conversion and from next run the code is working fine.
df = spark.read.format("delta").load("/mnt/dev/deltav2/X")
df= df.withColumn("LIFNR", df.LIFNR.cast("string"))
df.write.format('delta').option("overwriteSchema", "true").mode('overwrite').save("/mnt/dev/deltav2/X")
05-26-2023 02:25 AM
apparently the schema was not applied to your df.
You will have to check why (assuming the delta table col is indeed of type string). Without sharing code it is impossible to know.
05-26-2023 03:21 AM
@Vijay Kumar
Please share your code so that we can look and help you.
05-27-2023 01:07 AM
Hi @Vijay Kumar
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
05-29-2023 09:12 AM
Hi All,
the issue is resolved, i have executed column conversion and from next run the code is working fine.
df = spark.read.format("delta").load("/mnt/dev/deltav2/X")
df= df.withColumn("LIFNR", df.LIFNR.cast("string"))
df.write.format('delta').option("overwriteSchema", "true").mode('overwrite').save("/mnt/dev/deltav2/X")
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.