cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Dynamic Filtering Criteria for Data Streaming

leelee3000
New Contributor III
New Contributor III

One of the potential uses for DLT is a scenario where I have a large input stream of data and need to create multiple smaller streams based on dynamic and adjustable filtering criteria. The challenge is to allow non-engineering individuals to adjust these criteria without code modification. We can create a service or notebook around it that the individuals can use, but logically I am wondering if this makes sense for a DLT. In the context of DLT, is there a way to dynamically adjust filtering criteria for streaming data without redeploying the code? How can I achieve flexibility in filtering criteria adjustments?

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @leelee3000, I can provide a high-level approach to create a Spark DataFrame for streaming reads using Avro schemas from the Kafka schema registry.

 

Here's a general approach:

 

Retrieve the Avro schema: You can retrieve the Avro schema from the Kafka schema registry. This schema will be used to deserialize the data from Kafka.

Create a Spark DataFrame: Use the spark.readStream.format("kafka") method to create a DataFrame that reads from the Kafka topic. You must provide the Kafka bootstrap servers and the topic name as options.

Deserialize the data: Use the from_avro function to deserialize the data in the DataFrame. You'll need to provide the column to deserialize (the key or the value from the Kafka message) and the Avro schema.

 

Also, ensure that the necessary dependencies for Kafka and Avro are included in your Spark application. If you're using Databricks, they provide built-in support for Kafka and Avro

 

I hope this helps! Let me know if you have any other questions.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!