Spark pushdown filter not being respected on dbfs

raman
New Contributor II

‎12-18-2022 04:08 AM

I have a parquet files with a column g1 with schema

StructField(g1,IntegerType,true)

Now I have a query with filter on g1.

What's weird in the SQL viewer is that spark is loading all the rows from that file.

Even though in the physical plan I can see the "pushedFilter" condition being set.

This is created as a delta table on dbfs.

Any pointer on this would be helpful.

Thanks

Labels:
0 Kudos

Ajay-Pandey
Databricks MVP

‎12-18-2022 10:43 PM

Hi @Raman Gupta​ could you please share your code and physical plan for the same

Ajay Kumar Pandey
0 Kudos

raman
New Contributor II

‎12-19-2022 12:18 AM

Thanks @Ajay Pandey​ pls find attached the physical plan.

Query: Select identityMap, segmentMembership, _repo, workEmail, person, homePhone, workPhone, workAddress, personalEmail, homeAddress from final_segment_index_table_v2 where (g1 >= 128 AND g1 < 256).

Thanks

0 Kudos