Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
Showing results for 
Search instead for 
Did you mean: 

Are there any best practices on using Redis with structured streaming (for data enrichment) ?

Not applicable

Honored Contributor II

What would be the volume of data in Redis ?

Option 1: Read redis data into a dataframe and do a stream/static join

Option 2: Inside foreachBatch

  • Get redis connection
    • fetch lookup data by keys
    • perform the enrichment
    • close connection

New Contributor III

Redis offers various data structures such as strings, lists, sets, and hashes. Depending on your use case, select the appropriate data structure for storing the crm enrich data. For example, if you need to store key-value pairs, Redis hashes (HSET, HGET) can be a good choice. Define a consistent schema for the data stored in Redis. This helps in ensuring that the data retrieved from Redis aligns with the expected structure during enrichment. Consistency in data schema simplifies downstream processing. When retrieving data from Redis during structured streaming, avoid making individual requests for each record. Instead, consider using pipeline operations to send multiple commands in a batch. This can significantly reduce network round-trips and improve performance.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!