Internal Error During Spark SQL Phase Optimization – Possible Bug in Spark/Databricks Runtime

[julianner]

Bug Summary

Title: Spark SQL internal error when executing SQL code in the Azure Databricks SQL Editor

📋 Description

When executing SQL code in the Azure Databricks SQL Editor, a Spark SQL internal error is returned.

✅ Steps to Reproduce

1. Copy and paste the following code to a new query in Azure Databricks SQL Editor:

WITH

Synergy_Transactions AS

(

    SELECT T.TransactionDate

    , last_day(T.TransactionDate) as EOM_Transaction_Date

    , T.Status as TransactionStatus

    , T.ValueTotal as TransactionValueTotal

    FROM dbw_edp_prd_aue_01.silver02.pm_project_transaction_details T

    LEFT JOIN dbw_edp_prd_aue_01.silver02.pm_project_master P ON P.ProjectId = T.ProjectId AND P.IsCurrent = true

    LEFT JOIN dbw_edp_prd_aue_01.silver02.pm_project_stages S ON S.ProjectId = T.ProjectId AND S.IsCurrent = true

    WHERE P.ProjectNumber <> 'Internal'

    AND left(P.ProjectNumber,3) NOT IN ('INT','IAD','IPR')

    AND T.Status IN ('WIP','Draft invoice')

    AND S.FeeType NOT IN ('No billing')

    AND T.TransactionDate >= '2023-03-01'

    AND T.TransactionDate <= '2025-05-31'

    AND T.IsCurrent = true

)

SELECT T.EOM_Transaction_Date

, T.TransactionStatus

, SUM(T.TransactionValueTotal) as WIP

FROM Synergy_Transactions T

GROUP BY T.EOM_Transaction_Date

, T.TransactionStatus

1. Click the Run button to execute code

🔍 Expected Result

The query to run successfully and output aggregated dataset.

❌ Actual Result

The following error is experienced when executing the code:

[julianner]

Request: I would appreciate assistance with troubleshooting/resolving this problem

[saurabh18cs]

Check Databricks runtime version:
If possible, upgrade to the latest Databricks runtime, as internal errors are sometimes fixed in newer versions.