topic Spark pushdown filter not being respected on dbfs in Data Engineering

Spark pushdown filter not being respected on dbfs

raman — Sun, 18 Dec 2022 12:08:54 GMT

I have a parquet files with a column g1 with schema

StructField(g1,IntegerType,true)

Now I have a query with filter on g1.

What's weird in the SQL viewer is that spark is loading all the rows from that file.

Even though in the physical plan I can see the "pushedFilter" condition being set.

This is created as a delta table on dbfs.

Any pointer on this would be helpful.

Thanks

Re: Spark pushdown filter not being respected on dbfs

Ajay-Pandey — Mon, 19 Dec 2022 06:43:05 GMT

Hi @Raman Gupta could you please share your code and physical plan for the same

Re: Spark pushdown filter not being respected on dbfs

raman — Mon, 19 Dec 2022 08:18:35 GMT

Thanks @Ajay Pandey pls find attached the physical plan.

Query: Select identityMap, segmentMembership, _repo, workEmail, person, homePhone, workPhone, workAddress, personalEmail, homeAddress from final_segment_index_table_v2 where (g1 >= 128 AND g1 < 256).

Thanks