Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

julianner — Mon, 23 Jun 2025 07:02:36 GMT

Bug Summary

Title: Spark SQL internal error when executing SQL code in the Azure Databricks SQL Editor

📋 Description

When executing SQL code in the Azure Databricks SQL Editor, a Spark SQL internal error is returned.

✅ Steps to Reproduce

Copy and paste the following code to a new query in Azure Databricks SQL Editor:

WITH

Synergy_Transactions AS

(

SELECT T.TransactionDate

, last_day(T.TransactionDate) as EOM_Transaction_Date

, T.Status as TransactionStatus

, T.ValueTotal as TransactionValueTotal

FROM dbw_edp_prd_aue_01.silver02.pm_project_transaction_details T

LEFT JOIN dbw_edp_prd_aue_01.silver02.pm_project_master P ON P.ProjectId = T.ProjectId AND P.IsCurrent = true

LEFT JOIN dbw_edp_prd_aue_01.silver02.pm_project_stages S ON S.ProjectId = T.ProjectId AND S.IsCurrent = true

WHERE P.ProjectNumber <> 'Internal'

AND left(P.ProjectNumber,3) NOT IN ('INT','IAD','IPR')

AND T.Status IN ('WIP','Draft invoice')

AND S.FeeType NOT IN ('No billing')

AND T.TransactionDate >= '2023-03-01'

AND T.TransactionDate <= '2025-05-31'

AND T.IsCurrent = true

)

SELECT T.EOM_Transaction_Date

, T.TransactionStatus

, SUM(T.TransactionValueTotal) as WIP

FROM Synergy_Transactions T

GROUP BY T.EOM_Transaction_Date

, T.TransactionStatus

Click the Run button to execute code

🔍 Expected Result

The query to run successfully and output aggregated dataset.

❌ Actual Result

The following error is experienced when executing the code:

Re: Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

julianner — Mon, 23 Jun 2025 07:04:21 GMT

Request: I would appreciate assistance with troubleshooting/resolving this problem

Re: Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

saurabh18cs — Mon, 23 Jun 2025 07:59:06 GMT

Check Databricks runtime version:
If possible, upgrade to the latest Databricks runtime, as internal errors are sometimes fixed in newer versions.

topic Re: Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime in Warehousing & Analytics

Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

Re: Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

Re: Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime