topic Re: Dynamic Bloom Filters for Inner Joins in Get Started Discussions

Dynamic Bloom Filters for Inner Joins

tomvogel01 — Wed, 22 Jan 2025 06:43:58 GMT

I have a question regarding combining the use of Bloom filters with Liquid Clustering to further reduce the data read during a join/merge on top of dynamic file pruning. Testing both combined worked extremely well together for point queries. However having Bloom filters on a table removed dynamic file pruning entirely and lead to the entire table being read when doing a join/merge with and without Photon.

Do Bloom filters work along side dynamic file pruning? If so, any thoughts as to what might be going wrong?

Is there a plan to support such a functionality if not? If would be amazing to have it as it reduced the amount of data read by a factor of 20.

Re: Dynamic Bloom Filters for Inner Joins

tomvogel01 — Wed, 22 Jan 2025 09:18:52 GMT

Could you point me to the specific online resources that speak of this? My research has yielded very little in terms of guidance which is why I am reaching out here.

Re: Dynamic Bloom Filters for Inner Joins

NandiniN — Sun, 26 Jan 2025 11:49:08 GMT

We do not recommend Bloom filters Index on the Delta Tables as they have to be manually maintained.

If you prefer photon - please try predictive I/O with Liquid Clustering.