Louis_Frolio
Databricks Employee
Databricks Employee

a month ago

Hey @aharisaibabu , I did a little research and here is what I discovered:

Context first. The short answer is that the Snowflake Spark Connector still works and Snowflake still documents it, but it is no longer the strategic Databricks path. Treat it as tactical. Treat Lakehouse Federation plus Lakeflow Connect as the direction to build on.

Let me clear up the confusion first, because it matters. The disclaimer you quoted is not about the Query Federation bullet in your list. It is the archive banner on the Snowflake Spark Connector page itself (now at /archive/connectors/snowflake, marked Experimental). Databricks calls the old Spark connector approach "legacy query federation," which is a different thing from today's Lakehouse Federation (a Unity Catalog connection plus a foreign catalog). So the doc is not telling you to avoid federation. It is telling you to move off the Spark connector and onto Lakehouse Federation.

Now to your four questions.

  1. Is it still recommended and supported? It is usable, yes. Snowflake maintains the connector (net.snowflake:spark-snowflake). But Databricks has archived the page, flagged it Experimental, and points you to Lakehouse Federation. So: supported by Snowflake, de-emphasized by Databricks.

  2. Known limitations and concerns. A few to keep in mind:

  • Databricks will not test or troubleshoot it. You own the library version, and the bundled one can be stale on older runtimes like 10.4 LTS.
  • The config syntax differs between DBR 11.3 LTS and up versus 10.4 and below.
  • Write column order is not preserved, so you have to use columnmap.
  • Types can shift on round trips (INTEGER to NUMBER or DECIMAL).
  • Identifiers come back uppercase by default, so your schema gets uppercased.
  • Jobs running longer than 36 hours should use an external location to exchange data.
  • It is a direct format("snowflake") read, so it is not natively governed by Unity Catalog (no UC lineage or access control over the external data). Put credentials in secrets.

  1. For a custom-query pull, Spark connector or JDBC? If those are your only two choices, prefer the Spark connector over raw JDBC. It parallelizes through Snowflake stage unloads and pushes down by default, whereas plain JDBC is single-connection unless you manually set partitionColumn, lowerBound, upperBound, and numPartitions. But the real Databricks answer is "neither for the long term." Run your custom SQL against a Snowflake foreign catalog through Lakehouse Federation (filters, joins, aggregates, and window functions push down) and materialize into Delta with CTAS.

  2. The preferred long-term architecture. This is the documented decision rule:

  • For live, no-copy access, ad-hoc reporting, a proof of concept, exploratory ETL, or UC governance, use Lakehouse Federation. Snowflake is a supported source. It requires DBR 13.3 LTS and up, or pro/serverless SQL.
  • For persistent, scheduled, high-volume, lower-latency managed ingestion, use Lakeflow Connect. Databricks states that when a source supports both, Lakeflow Connect is preferred when volume and latency matter.

For your specific goal (custom-query incremental ingestion) here is the path that works today. There is not a dedicated GA Snowflake connector in Lakeflow Connect yet (it is listed as upcoming), but Lakeflow Connect's query-based connectors support all Lakehouse Federation data sources through foreign catalog ingestion. So you can do this in two steps: (1) create a Snowflake Lakehouse Federation connection and foreign catalog, then (2) build a query-based ingestion pipeline against it, using a cursor column (a single monotonically increasing column like updated_at or row_id) for incremental loads. That gets you serverless, UC-governed ingestion with no gateway and no staging volume.

Two honest caveats. Query-based connectors are in Preview so confirm the current release status before you put a production SLA on it. And for very large one-shot bulk extracts, the Spark connector's parallel stage unload can still beat Federation's JDBC reads. So benchmark if raw full-load throughput is your bottleneck, but weigh that against losing Databricks support and UC governance.

Takeaway: the Spark connector still works, but treat it as tactical only. Build on Lakehouse Federation now, and use Lakeflow Connect query-based ingestion through the Snowflake foreign catalog for your scheduled incremental loads. Don't anchor a new long-term architecture to the archived connector.

 

Cheers, Lou.

 

Sources I referenced for my research: