Re: Databricks Nested Json Flattening

mark_ott · ‎11-12-2025

You are encountering an AttributeError related to strip, which likely means that some entries for activity.value are not strings (maybe None or dicts) and your code expects all to be strings before calling .strip(). This kind of problem can arise if the JSON structure or types change unexpectedly, or if null/empty values are introduced in your data.

Problem Analysis

Here’s the problematic line:

python

sample_value_rdd_filtered = sample_value_rdd.filter(lambda x: x is not None and x.strip().startswith("{"))

If x is not a string (for example, None, a dictionary, or an integer), x.strip() will raise an AttributeError.
If some of your JSON data's activity.value is numeric, boolean, null, or an actual dictionary (not a stringified dictionary), this will break.

Why Did This Suddenly Start?

The underlying JSON structure in your data changed.
Rows were added with nulls or activity.value in a data type other than string.
Upstream process change or ingestion of new, unexpected formats.

Solution Approach

1. Make Your Filter More Robust

Make sure to only attempt .strip() on items that are truly strings:

python

sample_value_rdd_filtered = sample_value_rdd.filter(
    lambda x: isinstance(x, str) and x.strip().startswith("{")
)

This guards against None or non-string types and avoids AttributeError.

2. Debug Unexpected Types

To further debug, inspect a few values and their types. For example:

python

# Collect some values and their types for inspection
print(sample_value_rdd.take(5))
print([type(x) for x in sample_value_rdd.take(20)])

This will show which values are not strings.

3. Defensive Schema Inference

You may want to relax this logic further by handling actual dicts in addition to JSON strings, to protect future runs.

python

def is_json_string(x):
    return isinstance(x, str) and x.strip().startswith("{")

def is_json_dict(x):
    return isinstance(x, dict)

sample_value_rdd_filtered = sample_value_rdd.filter(lambda x: is_json_string(x) or is_json_dict(x))

4. Update downstream usage

When calling from_json, remember it only works with string columns—if the column contains dicts, you may need to cast or convert accordingly.

Summary of Fix

Change your filter to isinstance(x, str) before calling .strip().
Consider checking for dicts too, as some Spark operations may auto-convert JSON fields.
Check data source for upstream format changes.

Reference Example (Updated Filter)

python

sample_value_rdd_filtered = sample_value_rdd.filter(
    lambda x: isinstance(x, str) and x.strip().startswith("{")
)

Additional Suggestions

Log or sample problematic data to ensure the root cause is covered.
Write a test or assertion to check value types before Spark runs complex JSON parsing.
Add try/except in your filter (with a logger) to avoid silent failures.

This should resolve your error, and help protect against similar issues with dynamic, nested JSON in production pipelines.