cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

raman
by New Contributor II
  • 632 Views
  • 2 replies
  • 0 kudos

Spark pushdown filter not being respected on dbfs

I have a parquet files with a column g1 with schemaStructField(g1,IntegerType,true)Now I have a query with filter on g1.What's weird in the SQL viewer is that spark is loading all the rows from that file. Even though in the physical plan I can see th...

  • 632 Views
  • 2 replies
  • 0 kudos
Latest Reply
raman
New Contributor II
  • 0 kudos

Thanks @Ajay Pandey​ pls find attached the physical plan.Query: Select identityMap, segmentMembership, _repo, workEmail, person, homePhone, workPhone, workAddress, personalEmail, homeAddress from final_segment_index_table_v2 where (g1 >= 128 AND g1 <...

  • 0 kudos
1 More Replies
Labels