- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Monday
I am doing a case choice analysis on whether I should have my data in Snowflake to support my Customer workloads or have them migrated to Databricks to do the same.
What has been your experience especially in handling large volumes of data especially when dealing with real-time data coming to into your DWH?
My customers would like to see real-time insights almost in real-time with some of the most complex Data Science workloads.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
13 hours ago
Hi surya,
As per my experience, both are strong, but given your two needs, real-time ingestion plus heavy complex Data Science, I lean Databricks.
On real-time, Databricks treats streaming as first class. With Structured Streaming and Lakeflow Declarative Pipelines you can land high volume streams into Delta tables continuously and keep gold layers fresh with low latency. Snowflake has closed the gap with Snowpipe Streaming and Dynamic Tables, but it still feels more micro batch. If your customers mean seconds not minutes, Databricks gives more control.
On complex Data Science, this is where it pulls ahead for me. Native Spark, MLflow, feature store, and model serving in one place, so your scientists work right next to the data. Snowpark is improving fast, but for heavy custom ML at scale Databricks is more mature. On volume, both handle it, but Delta plus Photon has been cost effective for me, with less lock in.
One honest point, if it were mostly SQL and BI with light ML, Snowflake would be very compelling for its simplicity. But real-time plus complex DS on large data is exactly where the Lakehouse shines, since you do it all on one platform.
Last tip, agree with your customers on what real-time actually means. Streaming plus a fast serving layer gets you seconds. Truly sub second usually needs a dedicated serving store on top, either way.
Hope this helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
13 hours ago
Given the pace in the new AI era, which platform have the most inovations that brings value to customers clearly wins it. And we can see Databricks is releasing new features on weekly basis, they are clearly not behind.