cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Differentiate null values in Variant Data type

vgautam
New Contributor III

Hello, 

Based on the documentation here, in both scenarios below try_variant_get returns a null: 

  • If the object cannot be found
  • if the object cannot be cast 

How does one differentiate between the two scenarios?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Hi @vgautam ,

Please ignore the above code and use below code

%sql
WITH data AS (
  SELECT parse_json('{"key1": "value", "key2": 123}') AS variant_col
)
SELECT 
  -- Check if the key 'key3' exists and its type
  CASE 
    WHEN variant_get(variant_col, '$.key3') IS NULL THEN 'Key Not Found'
    WHEN try_variant_get(variant_col, '$.key3', 'string') IS NULL THEN 'Type Mismatch'
    ELSE 'Valid Cast'
  END AS key3_existence,

  -- Check if the key 'key1' exists and its type
  CASE
    WHEN variant_get(variant_col, '$.key1') IS NULL THEN 'Key Not Found'
    WHEN try_variant_get(variant_col, '$.key1', 'string') IS NULL THEN 'Type Mismatch'
    ELSE 'Valid Cast'
  END AS key1_type_check
FROM data;

View solution in original post

4 REPLIES 4

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @vgautam,

In the try_variant_get function, NULL is returned in two scenarios:

  1. Object Not Found: If the specified path does not exist in the JSON object.
  2. Invalid Cast: If the object at the specified path cannot be cast to the target type.

To differentiate between these two scenarios, you would need to handle the cases separately in your SQL logic. Unfortunately, the try_variant_get function itself does not provide a direct way to distinguish between these two scenarios. You may need to use additional logic or checks to determine the exact cause of the NULL value

@Alberto_Umana is it possible for you to elaborate on this please? What additional logic can be used to determine the exact cause? Could you please share an example?

Avinash_Narala
Valued Contributor II

Hi @vgautam ,

You can make use of try_variant_get and variant_get to further differentiate between the two null scenarios.

Please find the below code for your reference.

%sql
WITH data AS (
  SELECT parse_json('{"key1": "value", "key2": 123}') AS variant_col
)
SELECT 
  -- Check if the key 'key3' exists and its type
  CASE 
    WHEN try_variant_get(variant_col, '$.key3', 'int') IS NOT NULL THEN 'Valid Cast'
    WHEN try_variant_get(variant_col, '$.key3', 'string') IS NOT NULL THEN 'Type Mismatch'
    ELSE 'Key Not Found'
  END AS key3_existence,

  -- Check if the key 'key1' exists and its type
  CASE
    WHEN try_variant_get(variant_col, '$.key1', 'int') IS NOT NULL THEN 'Valid Cast'
    WHEN try_variant_get(variant_col, '$.key1', 'string') IS NOT NULL THEN 'Type Mismatch'
    ELSE 'Key Not Found'
  END AS key1_type_check
FROM data;
 

 

Hi @vgautam ,

Please ignore the above code and use below code

%sql
WITH data AS (
  SELECT parse_json('{"key1": "value", "key2": 123}') AS variant_col
)
SELECT 
  -- Check if the key 'key3' exists and its type
  CASE 
    WHEN variant_get(variant_col, '$.key3') IS NULL THEN 'Key Not Found'
    WHEN try_variant_get(variant_col, '$.key3', 'string') IS NULL THEN 'Type Mismatch'
    ELSE 'Valid Cast'
  END AS key3_existence,

  -- Check if the key 'key1' exists and its type
  CASE
    WHEN variant_get(variant_col, '$.key1') IS NULL THEN 'Key Not Found'
    WHEN try_variant_get(variant_col, '$.key1', 'string') IS NULL THEN 'Type Mismatch'
    ELSE 'Valid Cast'
  END AS key1_type_check
FROM data;

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group