filter push down into redis when querying using spark connector

almogg — Tue, 20 Jul 2021 09:41:50 GMT

I'm loading df from redis using this code:

df = (spark.read.format("org.apache.spark.sql.redis")
        .option("table", f"state_store_ready_to_sell")
        .option("key.column", "msid").option("infer.schema", "true").load()

and then i'm running filter , for example:

ready_to_sell = df.filter("msid in ('12321','12432')")

I looked at spark plan and spark does not push the msid filter to redis.

Which means that all redis records are loaded and filtered on spark memory (according to the sql tab is spark ui)

msid is key.column in redis of course.

How do i make spark push down the filter the fetch only the relevant records?

Thanks!

Almog

topic filter push down into redis when querying using spark connector in Data Engineering

filter push down into redis when querying using spark connector