Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2024 07:23 PM
I have a Personal cluster version 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12) and a SQL Warehouse in a databricks environment. When I use the following code to create a table in a catalog, it gives me different column types when run on the cluster vs the warehouse:
%sql
create or replace table [catalog].[schema].[test_table_name]
using delta
comment 'This has a comment'
as
select
id,
name as new_name,
created_date as new_created_date,
current_timestamp() as test_timestamp,
coalesce(name,'Replaced name') as test_coalesce_name,
coalesce(id,'-1') as test_coalesce_id,
coalesce(created_date,'2024-12-11') as test_coalesce_date
from (
select
cast(col1 as int) as id,
cast(col2 as string) as name,
cast(col3 as date) as created_date
from VALUES
(1, 'Alice', '2024-12-01'),
(2, 'Bob', '2024-12-02'),
(3, 'Charlie', '2024-12-03'),
(4, 'David', '2024-12-04'),
(5, 'Eve', '2024-12-05'),
(6, 'Frank', '2024-12-06'),
(7, 'Grace', '2024-12-07'),
(8, 'Hank', '2024-12-08'),
(9, 'Ivy', '2024-12-09'),
(10, 'Jack', '2024-12-10'),
(11, NULL, '2024-12-11'),
(NULL, 'NULL Values', NULL)
) as temp_tableWhen running on a SQL warehouse, the column types for the coalesce'd columns are resolved correctly. However, when running on a cluster, they are not resolved and are converted to strings. Is this expected behaviour?
Have tried on two different databricks environments and have the same result.