[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error

lurban — Thu, 09 May 2024 19:43:33 GMT

Hello,

I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a certain workflow, I receive the error below:

org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.

This is a workflow with 20+ tasks and fails at the final task step when executing this function. I tried the same function on an all-purpose cluster as well as running it in a smaller workflow with only 3 tasks and had no problems. However in this larger workflow, I am seeing the issue consistently after 3 + runs with no resolution.

I have loosely narrowed it to an issue with the Lakehouse Federation feature within unity catalog and potentially the fact that the workflow is larger. Has anyone run across this issue before and if so, what was the resolution?

Thanks in advance!

Re: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error

LindasonUk — Thu, 15 Aug 2024 13:27:35 GMT

I hit a similar error when trying to view FC data using a cluster with a lower Databricks Runtime.

Needs to be DBR 13.1 or higher.

topic Re: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error in Data Engineering

[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error

Re: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error