Databricks vs Snowflake Pyspark Performance
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
Hi experts, now the competition between cloud providers are fierce and brutal , I come across this post which compares Pyspark performance on databricks and Snowflake
https://l1nk.dev/1y33h0z although the metics tested are not rigorous and exhaustive, but comparative results has been shared. I want to know whether do we have similar comparisons on larger scale with wider set of benchmarks so when these discussions come up before prospective client, we have solid comparison resultset ready to share. Thank you in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
@Sam500 You can suggest PySpark on Databricks as Databricks is the original home of Apache Spark - the creators of Spark built Databricks optimizing it further and released it as a high performant platform for use. The Photon engine provides vectorized, native C++ execution that accelerates Spark workloads without any code changes delivering excellent performance on SQL and DataFrame operations. Further more, Liquid Clustering for adaptive data layout optimization and tight integration with Delta Lake for ACID transactions, time travel and unified batch/streaming are natively tuned for Spark workloads. Other platforms cannot match it. More details here
Databricks outperforms other platforms with a unified lake house - one platform for data engineering, BI, ML/AI, Genie, Agents and streaming without data silos or movement. Databricks supports open formats (Delta, Parquet, Iceberg) with no vendor lock-in. With Lakeflow for unified ingestion and pipeline orchestration, Genie for conversational AI, Agents, BI, Mosaic AI for end-to-end model development and serving, Unity Catalog for unified governance across data and AI and Lake base for operational workloads - capabilities other platforms cannot match in a single platform. More details here
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
Thank you @balajij8 for reply. I agree with first part of you answer, although the link shared by you talks about different use case. As I said before, the qualifiable metrics should entirely dealt with Databricks vs Snowflake Pyspark Performance , cost metrics and scalability aspects. I would appreciate if you could share anything on this . Thanks again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
Good question. A standardized benchmark covering performance, cost, scalability, and different workloads would make these comparisons much more useful for real client discussions.
