Hello,
I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a certain workflow, I receive the error below:
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
This is a workflow with 20+ tasks and fails at the final task step when executing this function. I tried the same function on an all-purpose cluster as well as running it in a smaller workflow with only 3 tasks and had no problems. However in this larger workflow, I am seeing the issue consistently after 3 + runs with no resolution.
I have loosely narrowed it to an issue with the Lakehouse Federation feature within unity catalog and potentially the fact that the workflow is larger. Has anyone run across this issue before and if so, what was the resolution?
Thanks in advance!
