Databricks Community

Dhruv-22

I asked a question recently about VOID in STRUCT failing to cast to VARIANT, whereas a VOID column is castable to VARIANT. LinkEarlier I thought it might be an error since cast was possible somehow. But the community members replied that this is the ...

Dhruv-22

Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...

Dhruv-22 · 04-28-2026

I've a table with the following profiling settings{ "status": "MONITOR_STATUS_ACTIVE", "profile_metrics_table_name": "edw_prd_aen.silver.fct_retail_permit_profile_metrics", "drift_metrics_table_name": "edw_prd_aen.silver.fct_retail_permit_drift_me...

Dhruv-22 · 01-12-2026

The following is a minimal reproducible example of what I'm facing right now.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.test_table ( id INT ); INSERT INTO edw_nprd_aen.bronze.test_table VALUES (1); SELECT * FROM edw_nprd_aen.bronze.test_tab...

Dhruv-22 · 01-09-2026

I want to raise a feature request as follows.Currently, in the Automatic schema evolution for merge when a column is not present in the source dataset it is not changed in the target dataset. For e.g.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.t...

Dhruv-22

Hi @ThomazNeto Thanks for the pointers — the SPARK-51576 and the context was genuinely useful for chasing this down further.However, there are few points I would like to bring upYour 'conflating SQL NULL with variant null' and 'manufacturing "a value...

Dhruv-22

Hi @AbhilashNagilla Thanks for the suggestions. It helps in ways to work around it.But also, I'm trying to understand the underlying engine behavior because there seems to be an inconsistency in how VOID is handled:TO_VARIANT_OBJECT accepts VOID in A...

Dhruv-22

Hi @balajij8 I know about the drawbacks of VOID and the use of VOID column while converting to VARIANT is not deliberate. It is just that the data for the entire column came empty, so while reading spark converted it to VOID. But that is not my quest...

Dhruv-22 · 03-12-2026

Hey @SteveOstrowski Thanks for the effort and suggestion. I have given the feedback via databricks feedback. I don't have any support though.

Dhruv-22 · 03-10-2026

Hi @SteveOstrowski Thanks for replying. I got the process of how databricks processes the schema evolution command internally. It helps a lot.As you mentioned there are 2 steps. First, is the explicit set statement which I wrote. This statement is pr...

Databricks Community

User Stats

User Activity

(Design Decision) Why VOID in STRUCT is not castable to VARIANT?

VOID column inside STRUCT fails to cast to VARIANT

ProfilingError: SPARK_ERROR. Spark encountered an error while refreshing metrics.

Merge with schema evolution fails because of upper case columns

Feature request: Allow to set value as null when not present in schema evolution

Re: (Design Decision) Why VOID in STRUCT is not castable to VARIANT?

Re: VOID column inside STRUCT fails to cast to VARIANT

Re: VOID column inside STRUCT fails to cast to VARIANT

Re: Merge with schema evolution fails because of upper case columns

Re: Merge with schema evolution fails because of upper case columns