java.util.NoSuchElementException: key not found
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-17-2022 03:08 PM
Hello,
We are using a Azure Databricks with Standard DS14_V2 Cluster with Runtime 9.1 LTS, Spark 3.1.2 and Scala 2.12 and facing the below issue frequently when running our ETL pipeline. As part of the operation that is failing there are several joins happening with delta tables and the output of that is being written into another delta table.
There was a similar issue reported earlier and was fixed as part of DB runtime 9.0 (unsupported) : [SPARK-34000] ExecutorAllocationListener threw an exception java.util.NoSuchElementException - ASF...
Here is the documentation for that : Databricks Runtime 9.0 (Unsupported) - Azure Databricks | Microsoft Docs
Is the below exception an unrelated issue from the one reported in the bug mentioned above and should I be logging another bug for this to be fixed? We are facing this issue every few runs of our daily job. Any help or pointers would be greatly appreciated
Here is the exception with the stack trace : I have uploaded a file with the full stack trace because of the size limit
An error occurred while calling o74209.insertInto.
: java.util.NoSuchElementException: key not found: Project [none#1 AS #0, none#4, none#18 AS #1, none#13, none#12, none#3, CASE WHEN isnull(none#19) THEN -25567 ELSE cast(gettimestamp(none#19, yyyy-MM-dd, Some(Etc/UTC), false) as date) END AS #2, none#20, none#0 AS #3, (none#3 = '') AS #4]
+- Relation[none#0,none#1,none#2,none#3,none#4,none#5,none#6,none#7,none#8,none#9,none#10,none#11,none#12,none#13,none#14,none#15,none#16,none#17,none#18,none#19,none#20,none#21,none#22,none#23,... 5 more fields] parquet
at scala.collection.MapLike.default(MapLike.scala:235)
at scala.collection.MapLike.default$(MapLike.scala:234)
at scala.collection.AbstractMap.default(Map.scala:63)
at scala.collection.MapLike.apply(MapLike.scala:144)
at scala.collection.MapLike.apply$(MapLike.scala:143)
at scala.collection.AbstractMap.apply(Map.scala:63)
at com.databricks.sql.transaction.tahoe.stats.PrepareDeltaScan$$anonfun$prepareDeltaScanParallel$1.applyOrElse(PrepareDeltaScan.scala:229)
at com.databricks.sql.transaction.tahoe.stats.PrepareDeltaScan$$anonfun$prepareDeltaScanParallel$1.applyOrElse(PrepareDeltaScan.scala:227)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anon$2.apply(QueryPlan.scala:545)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anon$2.apply(QueryPlan.scala:541)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anon$2.applyOrElse(QueryPlan.scala:541)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:484)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:86)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:484)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:29)
Thanks & Regards,
Hitesh
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2022 01:53 AM
Hey man,
Please use these configuration in your cluster and it will work,
spark.sql.storeAssignmentPolicy LEGACY
spark.sql.parquet.binaryAsString true
spark.speculation false
spark.sql.legacy.timeParserPolicy LEGACY
if it wont work let me know what problem you are facing there,we can discuss