topic in-home built predictive optimization in Data Engineering

in-home built predictive optimization

noorbasha534 — Mon, 21 Jul 2025 21:10:50 GMT

Hello all

Has anyone attempted to look at the internals of predictive optimization and built an in-home solution mimicking its functionality? I understood that there are no plans from Databricks to roll-out this feature for external tables, and hence, we were thinking to gather on our own the telemetry of frequently used columns and use that information for liquid clustering and gathering stats....

On the other hand, if Databricks can open source it, that would be really helpful...

Re: in-home built predictive optimization

lingareddy_Alva — Mon, 21 Jul 2025 21:27:34 GMT

Hi @noorbasha534

You're touching on a really interesting area! While Databricks hasn't open-sourced predictive optimization,
there have been some community efforts and approaches to build similar functionality:

Community Efforts:

Yes, some teams build DIY solutions using Spark query logs and custom listeners
Focus on liquid clustering column selection and automated stats collection
No full open-source clone exists yet

Common Approaches:
Parse Spark History Server logs for column usage patterns
Custom EventListeners to capture query telemetry
Heuristic-based optimization scheduling

Reality Check:
Targeted solutions (clustering hints, stats automation) are feasible
Full predictive optimization replication is complex
Databricks hasn't indicated plans to open-source it

Bottom Line: Build incrementally - start with query pattern analysis for liquid clustering decisions, then expand based on ROI.

Re: in-home built predictive optimization

szymon_dybczak — Tue, 22 Jul 2025 05:55:08 GMT

Hi @noorbasha534 ,

That’s a really cool idea and definitely shows initiative - but realistically, it might not be worth the effort. There’s a lot of engineering going on under the hood that would be tough to replicate in-house.

Collecting telemetry and using it for things like liquid clustering and stats gathering could work to some extent, but the effort required to build and maintain something similar would likely outweigh the benefits, especially given how deeply integrated and optimized the native solution is.
If you have external tables I would just take care of regular maintenance of the tables (etc. like running optimize/ vacuum regulary).

Would be awesome if Databricks open-sourced it, though - totally agree with you there.

Re: in-home built predictive optimization

noorbasha534 — Tue, 22 Jul 2025 06:53:43 GMT

@szymon_dybczak since liquid clustering only allows 4 columns to be set for now, I think I can go blindly with the primary keys here. In our case, we have wide tables with 300+ columns, and users are querying on columns that are not in the first 32 positions for which we are gathering stats, and the stats gathering is not really helping us.

Re: in-home built predictive optimization

noorbasha534 — Tue, 22 Jul 2025 11:48:53 GMT

@LinlinH thanks for the details. Can you please share any Github link where the community work is put so I can verify if any code can be re-used...

Re: in-home built predictive optimization

alsetr — Tue, 22 Jul 2025 13:58:23 GMT

HI, @noorbasha534
If you use DBR 13.3+, you can specify columns for which you would like to collect statistics with delta.dataSkippingStatsColumns

https://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping