Dynamic Bloom Filters for Inner Joins
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2025 10:43 PM
I have a question regarding combining the use of Bloom filters with Liquid Clustering to further reduce the data read during a join/merge on top of dynamic file pruning. Testing both combined worked extremely well together for point queries. However having Bloom filters on a table removed dynamic file pruning entirely and lead to the entire table being read when doing a join/merge with and without Photon.
Do Bloom filters work along side dynamic file pruning? If so, any thoughts as to what might be going wrong?
Is there a plan to support such a functionality if not? If would be amazing to have it as it reduced the amount of data read by a factor of 20.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2025 01:18 AM
Could you point me to the specific online resources that speak of this? My research has yielded very little in terms of guidance which is why I am reaching out here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2025 03:49 AM
We do not recommend Bloom filters Index on the Delta Tables as they have to be manually maintained.
If you prefer photon - please try predictive I/O with Liquid Clustering.
