Spark Settings in SQL Warehouse
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2023 03:30 AM
I'm running a query, trying to parse a string into a map, and I get the following error;
org.apache.spark.SparkRuntimeException: Duplicate map key was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so that the key inserted at last takes precedence.No worries, so I try to alter this setting;
set spark.sql.mapKeyDedupPolicy=LAST_WINAnd get the following error;
TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: Configuration spark.sql.mapKeyDedupPolicy is not available.:155:154, This works when using a general purpose cluster, but doesn't work when using a SQL Warehouse. So how do I change this setting in a SQL Warehouse cluster? And if that's not possible, how do I get around this error?x
Labels:
- Labels:
-
Key
-
SQL
-
Sql Warehouse