The same bug is affecting me, but only when using Databricks runtime 14.3 LTS single user cluster.

I'm using row_number() on string columns, which should result in an integer. However, Spark internally seems to raise an error on not being able to convert Long and Int.

Example code:

WITH A AS (
  SELECT
  --opportunityid is a string
    ROW_NUMBER() OVER(ORDER BY tbl_a.opportunityid) AS PK_A
  FROM 20_silver_crmeiw.opportunities AS tbl_a
)

, B AS (
  SELECT
  --mcw_contractid is a string
    ROW_NUMBER() OVER(ORDER BY tbl_b.mcw_contractid) AS PK_B
  FROM 20_silver_crmeiw.mcw_contracts AS tbl_b
)

, C AS (
  SELECT
    PK_A
  FROM A

  UNION
  
  SELECT
    PK_B
  FROM B
)

SELECT
  *
FROM A
LEFT JOIN C
ON 1=1

Without the join, this code is fine:

WITH A AS (
  SELECT
  --opportunityid is a string
    ROW_NUMBER() OVER(ORDER BY tbl_a.opportunityid) AS PK_A
  FROM 20_silver_crmeiw.opportunities AS tbl_a
)

, B AS (
  SELECT
  --mcw_contractid is a string
    ROW_NUMBER() OVER(ORDER BY tbl_b.mcw_contractid) AS PK_B
  FROM 20_silver_crmeiw.mcw_contracts AS tbl_b
)

SELECT
PK_A
FROM A

UNION

SELECT
PK_B
FROM B

Querying SELECT typeof(PK_A) FROM A and SELECT typeof(PK_B) FROM B both return 'int'.