- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2023 01:12 AM
I am have imported a csv file using spark.read method, i have used custom schema and declared the type of the column as string.
i have delta table and the type of the column in the table is also string.
I am getting failed to merge fields errors in spark.write.format('delta') command,
can you please suggest.
- Labels:
-
Data
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2023 02:25 AM
apparently the schema was not applied to your df.
You will have to check why (assuming the delta table col is indeed of type string). Without sharing code it is impossible to know.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2023 03:21 AM
@Vijay Kumar
- Check your schema of the table using DESCRIBE TABLE and the dataframe you loaded using df.printSchema() and compare which column has a mismatch. You need to see if your custom schema is matching the table schema.
- MergeSchema will support only the schemas that it can typecast.
- If you want to completely recreate the table schema with the ones of dataframes, you can use overwriteSchema option.
Please share your code so that we can look and help you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2023 01:07 AM
Hi @Vijay Kumar
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-29-2023 09:12 AM
Hi All,
the issue is resolved, i have executed column conversion and from next run the code is working fine.
df = spark.read.format("delta").load("/mnt/dev/deltav2/X")
df= df.withColumn("LIFNR", df.LIFNR.cast("string"))
df.write.format('delta').option("overwriteSchema", "true").mode('overwrite').save("/mnt/dev/deltav2/X")
