topic Re: Are there any best practices on using Redis with structured streaming (for data enrichment) ? in Administration & Architecture

Are there any best practices on using Redis with structured streaming (for data enrichment) ?

Anonymous — Tue, 18 Mar 2025 16:51:02 GMT

Re: Are there any best practices on using Redis with structured streaming (for data enrichment) ?

sajith_appukutt — Wed, 23 Jun 2021 06:33:34 GMT

What would be the volume of data in Redis ?

Option 1: Read redis data into a dataframe and do a stream/static join

Option 2: Inside foreachBatch

Get redis connection
- fetch lookup data by keys
- perform the enrichment
- close connection

Re: Are there any best practices on using Redis with structured streaming (for data enrichment) ?

Joe_Suarez — Thu, 29 Jun 2023 13:42:09 GMT

Redis offers various data structures such as strings, lists, sets, and hashes. Depending on your use case, select the appropriate data structure for storing the crm enrich data. For example, if you need to store key-value pairs, Redis hashes (HSET, HGET) can be a good choice. Define a consistent schema for the data stored in Redis. This helps in ensuring that the data retrieved from Redis aligns with the expected structure during enrichment. Consistency in data schema simplifies downstream processing. When retrieving data from Redis during structured streaming, avoid making individual requests for each record. Instead, consider using pipeline operations to send multiple commands in a batch. This can significantly reduce network round-trips and improve performance.