cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error

lurban
New Contributor

Hello,

I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a certain workflow, I receive the error below: 

org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.

This is a workflow with 20+ tasks and fails at the final task step when executing this function. I tried the same function on an all-purpose cluster as well as running it in a smaller workflow with only 3 tasks and had no problems. However in this larger workflow, I am seeing the issue consistently after 3 + runs with no resolution. 

I have loosely narrowed it to an issue with the Lakehouse Federation feature within unity catalog and potentially the fact that the workflow is larger. Has anyone run across this issue before and if so, what was the resolution? 

 

Thanks in advance! 

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @lurban,

  1.  

Hopefully, one of these approaches will lead you toward a resolution! 

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!