topic I'm using the Redshift data source to load data into spark SQL data frames. However, I'm not seeing predicate push down for my queries ran on Redshift - is that expected? in Data Engineering

I'm using the Redshift data source to load data into spark SQL data frames. However, I'm not seeing predicate push down for my queries ran on Redshift - is that expected?

sajith_appukutt — Wed, 09 Jun 2021 08:28:20 GMT

I was expecting filter operations to be pushed down to Redshift by the optimizer. However, the entire dataset is getting loaded from Redshift.

Re: I'm using the Redshift data source to load data into spark SQL data frames. However, I'm not seeing predicate push down for my queries ran on Redshift - is that expected?

sajith_appukutt — Tue, 22 Jun 2021 01:02:06 GMT

The Spark driver for Redshift pushes the following operators down into Redshift:

Filter
Project
Sort
Limit
Aggregation
Join

However, it does not support expressions operating on dates and timestamps today. If you have a similar requirement, please add a feature request via https://docs.databricks.com/resources/ideas.html