How can I preserve the data type of the delta tables while writing to Azure blob storage ?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-11-2024 09:40 PM
I am writing a file using this but the data type of columns get changed while reading.
df.write.format("com.crealytics.spark.excel").option("header", "true").mode("overwrite").save(path)
Due to this I have to manual change every time as I can't change the datatype of my target. Is there any way I can write my file with the same datatypes?
[DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'quantity_volume' and 'quantity_volume' SQLSTATE: 22005
File <command-788927452842747>, line 6 4 df.printSchema() 5 df.count() ----> 6 df.write.mode("overwrite").saveAsTable("cc.pg_vendor_product_id_hierarchy")
File /databricks/spark/python/pyspark/errors/exceptions/captured.py:230, in capture_sql_exception.<locals>.deco(*a, **kw) 226 converted = convert_exception(e.java_exception) 227 if not isinstance(converted, UnknownException): 228 # Hide where the exception came from that shows a non-Pythonic 229 # JVM exception message. --> 230 raise converted from None 231 else: 232 raise
