Hi,
I am having an issue of loading source data into a delta table/ unity catalog. The error we are recieving is the following:
grpc_message:"[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Exceeds char/varchar type length limitation. Failed check: (isnull(\'metric_name) OR (length(\'metric_name) <= 0))
We get this issue when executing the line below:
source_df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(destination_mount_filepath)
The background of this is we are loading raw data into a delta table in mounted storage.
I have tried unmounting/remounting storage and recreating the container in storage account. I have checked and validated the data and there are no rows that would fall to this match. I have tried dropping the Unity Catalog tables and using the dbutils to remove the old data behind the Catalog tables to no avail.