Spark Settings in SQL Warehouse

najmead
Contributor

I'm running a query, trying to parse a string into a map, and I get the following error;

org.apache.spark.SparkRuntimeException: Duplicate map key  was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so that the key inserted at last takes precedence.

No worries, so I try to alter this setting;

set spark.sql.mapKeyDedupPolicy=LAST_WIN

And get the following error;

TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: Configuration spark.sql.mapKeyDedupPolicy is not available.:155:154, 

This works when using a general purpose cluster, but doesn't work when using a SQL Warehouse. So how do I change this setting in a SQL Warehouse cluster? And if that's not possible, how do I get around this error?x