For one of badge completion, it was mandatory to complete a Spark Streaming Demo Practice.

Due to the absence of a Kafka broker setup required for the demo practice, I configured a Confluent Kafka cluster and made several modifications to the Spark script provided by DBDemos streaming-sessionization to ensure compatibility with the Confluent Kafka cluster. As a result, I successfully ingested data from the Kafka topic into the bronze table.

Upon initiating the data load from the bronze table to the silver table, I observed an anomaly where the code executed successfully, and the transaction log indicated a 'Streaming_update' in delta table logs. However, the table did not contain any data. Further investigation revealed that the from_json function was not parsing the values correctly, despite the schema I was passing to function was as expected or in script.

Step By Step Resolving 

1. Understand the schema of value column from Kafka topic. Using schema_of_json function we can dynamically extract schema from stringified JSON value.

2. Now trying from_json with json_schema to convert value column into individual key columns, where key-value pair parse but values were becoming null.

Also, checked whether JSON is valid by using python json lib, which parsed it without any error.

3. Finally, resolving the issue with few simple strings cleaning as mentioned below, it worked.

a) Replacing the leading and trailing double quotes from string value
b) Replacing back-slash with empty-string

I'm not sure whether it's expected behavior of from_json function or it should be fixed.

#DataEngineering #StreamingSessionization #dbdemos #SparkStructuredStreaming

Regards,
Hari Prasad