- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2025 10:17 AM
Hi 5UDO, No worries at all, and thanks for the clarification—it actually makes your results even more insightful! You're right in noting that Bloom filters on a hashed column can help narrow down files, but they don’t always guarantee fast reads if the underlying data layout isn’t tight. Also, as you observed, scaling up the warehouse size often improves performance on less optimized tables, but has diminishing returns once your data is already well-structured. You're spot on about Z-Ordering generally being better for read-heavy workloads, especially for highly selective queries like yours, while Liquid Clustering shines more for write-heavy or schema-evolving tables. That said, Liquid Clustering can outperform Z-Ordering in scenarios where data changes frequently or where file compaction and cluster-aware sorting offer better long-term layout maintenance—so it's worth trying if you want a more hands-off clustering method.
As for your lookup use case, building a materialized index table could help—you’d create a compact Delta table with just the hashed_id, maybe the original id, and the necessary dimensions for joins, and Z-Order that tiny table for quick binary-style lookups. It’s a pattern similar to building a pre-joined lookup table. Since any person could be queried at random, making lookup predictable and cacheable at the storage layer becomes key.
You’ve already done great testing—at this point, it’s about fine-tuning your access pattern. Let me know if you'd like help designing that index table or trying a hybrid LC + Z-Order setup to compare results!
Regards,
Brahma