Dynamic Filtering Criteria for Data Streaming

leelee3000 — Tue, 12 Dec 2023 21:44:50 GMT

One of the potential uses for DLT is a scenario where I have a large input stream of data and need to create multiple smaller streams based on dynamic and adjustable filtering criteria. The challenge is to allow non-engineering individuals to adjust these criteria without code modification. We can create a service or notebook around it that the individuals can use, but logically I am wondering if this makes sense for a DLT. In the context of DLT, is there a way to dynamically adjust filtering criteria for streaming data without redeploying the code? How can I achieve flexibility in filtering criteria adjustments?

topic Dynamic Filtering Criteria for Data Streaming in Data Engineering

Dynamic Filtering Criteria for Data Streaming