cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Snowflake OR Databricks

suryaprayaga
New Contributor

I am doing a case choice analysis on whether I should have my data in Snowflake to support my Customer workloads or have them migrated to Databricks to do the same.

What has been your experience especially in handling large volumes of data especially when dealing with real-time data coming to into your DWH? 

My customers would like to see real-time insights almost in real-time with some of the most complex Data Science workloads.

suryaprayaga
1 ACCEPTED SOLUTION

Accepted Solutions

Brahmareddy
Esteemed Contributor II

Hi surya,

As per my experience, both are strong, but given your two needs, real-time ingestion plus heavy complex Data Science, I lean Databricks.

On real-time, Databricks treats streaming as first class. With Structured Streaming and Lakeflow Declarative Pipelines you can land high volume streams into Delta tables continuously and keep gold layers fresh with low latency. Snowflake has closed the gap with Snowpipe Streaming and Dynamic Tables, but it still feels more micro batch. If your customers mean seconds not minutes, Databricks gives more control.

On complex Data Science, this is where it pulls ahead for me. Native Spark, MLflow, feature store, and model serving in one place, so your scientists work right next to the data. Snowpark is improving fast, but for heavy custom ML at scale Databricks is more mature. On volume, both handle it, but Delta plus Photon has been cost effective for me, with less lock in.

One honest point, if it were mostly SQL and BI with light ML, Snowflake would be very compelling for its simplicity. But real-time plus complex DS on large data is exactly where the Lakehouse shines, since you do it all on one platform.

Last tip, agree with your customers on what real-time actually means. Streaming plus a fast serving layer gets you seconds. Truly sub second usually needs a dedicated serving store on top, either way.

Hope this helps.

View solution in original post

2 REPLIES 2

Brahmareddy
Esteemed Contributor II

Hi surya,

As per my experience, both are strong, but given your two needs, real-time ingestion plus heavy complex Data Science, I lean Databricks.

On real-time, Databricks treats streaming as first class. With Structured Streaming and Lakeflow Declarative Pipelines you can land high volume streams into Delta tables continuously and keep gold layers fresh with low latency. Snowflake has closed the gap with Snowpipe Streaming and Dynamic Tables, but it still feels more micro batch. If your customers mean seconds not minutes, Databricks gives more control.

On complex Data Science, this is where it pulls ahead for me. Native Spark, MLflow, feature store, and model serving in one place, so your scientists work right next to the data. Snowpark is improving fast, but for heavy custom ML at scale Databricks is more mature. On volume, both handle it, but Delta plus Photon has been cost effective for me, with less lock in.

One honest point, if it were mostly SQL and BI with light ML, Snowflake would be very compelling for its simplicity. But real-time plus complex DS on large data is exactly where the Lakehouse shines, since you do it all on one platform.

Last tip, agree with your customers on what real-time actually means. Streaming plus a fast serving layer gets you seconds. Truly sub second usually needs a dedicated serving store on top, either way.

Hope this helps.

DoTA
New Contributor III

Given the pace in the new AI era, which platform have the most inovations that brings value to customers clearly wins it. And we can see Databricks is releasing new features on weekly basis, they are clearly not behind.