Dynamic Filtering Criteria for Data Streaming

[leelee3000]

One of the potential uses for DLT is a scenario where I have a large input stream of data and need to create multiple smaller streams based on dynamic and adjustable filtering criteria. The challenge is to allow non-engineering individuals to adjust these criteria without code modification. We can create a service or notebook around it that the individuals can use, but logically I am wondering if this makes sense for a DLT. In the context of DLT, is there a way to dynamically adjust filtering criteria for streaming data without redeploying the code? How can I achieve flexibility in filtering criteria adjustments?

[Kaniz]

Hi @leelee3000, I can provide a high-level approach to create a Spark DataFrame for streaming reads using Avro schemas from the Kafka schema registry.

Here's a general approach:

Retrieve the Avro schema: You can retrieve the Avro schema from the Kafka schema registry. This schema will be used to deserialize the data from Kafka.

Create a Spark DataFrame: Use the spark.readStream.format("kafka") method to create a DataFrame that reads from the Kafka topic. You must provide the Kafka bootstrap servers and the topic name as options.

Deserialize the data: Use the from_avro function to deserialize the data in the DataFrame. You'll need to provide the column to deserialize (the key or the value from the Kafka message) and the Avro schema.

Also, ensure that the necessary dependencies for Kafka and Avro are included in your Spark application. If you're using Databricks, they provide built-in support for Kafka and Avro. 

I hope this helps! Let me know if you have any other questions.