- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2025 08:39 AM
Hi Brahmareddy and AndrewN,
Thank you on your answers.
I first need to apologize as I accidentally wrote wrong that I got 270ms with hashing the date of birth, surname and name and then using the z ordering.
I actually achieved around 290ms with hashing the date of birth, surname and name and then using bloom indexing on the hash value column.
@Brahmareddy
For now I tried to use 2X Large warehouse, but I only saw better performance on unoptimized tables.
Tables which were running great had marginal improvement, maybe 10-20ms.
I even tried to run the same queries 2 times to see how fast is Local Cache for queries, but even that wasn't always faster than 200ms.
But even if execution time was always under 200ms I would need some different approach, since there is equal chance that any person could be queried at any time.
I tried to run ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS but there was no improvement.
I assume because there were no changes on the table since it was created.
materialize a lightweight index table with pre-hashed or encoded keys just for lookup purposes
This is still something I need to test. But I'm also not sure what is the best way to do it in Databricks.
@AndrewN
I didn't yet tried Liquid Clustering, but, from what I've read, in some cases Z ordering is faster for reading data, while Liquid Clustering is faster for writing data.
Are there some cases where you got better reading speeds with LC than with Z ordering?