Feature Store Benchmarks

[FedeRaimondi]

We are currently planning to create feature tables to serve machine learning models in our organization.

I am struggling to find interesting benchmarks on Databricks Feature Store performances vs using directly Delta Tables. It would also be interesting the different results for batch inference and usage of online feature store for real time inference.

Throwing the topic in the community to see if someone in Databricks or other clients have tried running some tests. 

[WiliamRosa]

We’re also exploring this internally and found very limited public benchmarks comparing Databricks Feature Store to directly using Delta Tables. That said, the open-source project featurestore-benchmarks provides a framework to evaluate offline and online feature store performance across platforms, which could be adapted for Databricks:
https://github.com/featurestoreorg/featurestore-benchmarks

Additionally, Hopsworks published some academic benchmarks comparing their feature store to Databricks, SageMaker, and Vertex AI. While results may not generalize fully, they provide useful performance reference points:
https://www.hopsworks.ai/news/redefining-feature-stores-with-class-leading-performance

From community discussions, Delta Tables may be sufficient for batch inference, but Feature Store provides added value for point-in-time joins, versioning, and online inference:
https://www.reddit.com/r/mlops/comments/14fj1o7
https://www.reddit.com/r/mlops/comments/17p0w7h

We’re considering setting up our own benchmarks using these tools. Would be great to hear if others have done similar testing on Databricks.