Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-02-2020 06:42 AM
You have to set the maxStrLength option to a value bigger than the longest string in your source data
synapseDF.write \
.format("com.databricks.spark.sqldw") \
.option("url", connStr ) \
.mode( "append" ) \
.option("tempDir", synapse_tempDir ) \
.option("forwardSparkAzureStorageCredentials", "true") \
.option("maxStrLength", "1024" ) \
.option("dbTable", synapse_targetschema + "." + synapse_targettable ) \
.save()
Synapse creates a temporary external table while loading data with Polybase, so even if you create your target table with columns of the appropriate width, you can still get truncation errors from this temp table if you don't set maxStrLength