Lakehouse Federation (Snowflake) — large query results fail to download from internal stage

[margarita_shir]

We're hitting a consistent failure with Lakehouse Federation to Snowflake where large result sets fail during the result-chunk download, while small results work. Looking for help isolating whether this is the Databricks Snowflake connector or a config issue on our side.

Setup
- Databricks on AWS, customer-managed VPC, us-east-1.
- Snowflake foreign catalog; account reached over PrivateLink (…privatelink.snowflakecomputing.com).
- Reproduced on a SQL warehouse, both Current and Preview channels.

Behavior (size-dependent):
- SELECT * FROM <table> LIMIT 5 → returns fine.
- SELECT * FROM <table> LIMIT 100 → hangs, then errors.

Error:
JDBC driver internal error: Max retry reached for the download of chunk#0
(Total chunks: 1) retry: 7, error: net.snowflake.client.jdbc.SnowflakeSQLException:
JDBC driver encountered communication error. Message: Exception encountered for
HTTP request: sfc-va2-ds1-18-customer-stage.s3.amazonaws.com:443 failed to respond.
at net.snowflake.client.jdbc.RestRequest.executeWithRetries(RestRequest.java:1082)

What we've established:
- The query succeeds on Snowflake's side (confirmed via their query history — it scans and returns rows); the client then cancels after it can't pull the result chunk. So this is the client-side download of result chunks from Snowflake's internal S3 stage, not permissions or query logic.
- Small results come back inline and work; large results stage to S3 and the connector fails on chunk#0. That isolates it to the internal-stage result-chunk download path.
- Network is not blocking it: from the compute, a direct request to the stage bucket (sfc-va2-ds1-18-customer-stage.s3.amazonaws.com) returns a clean HTTP 403 (reached S3, S3 responded). We have an S3 gateway endpoint with a wide-open policy, and the subnets route the us-east-1 S3 prefix list to it.
- Not a driver-version issue: it reproduces on the Preview channel (newer connector) as well as Current.

Is this a known issue with the Snowflake federation connector on a PrivateLink account connection + public-S3 internal stages, and is there a recommended fix or workaround?

[pradeep_singh]

This is most likely a network-path issue, not a query or permissions problem.

Small results work because they are returned directly. Larger results are downloaded from Snowflake’s S3 stage, and that download is failing. Snowflake PrivateLink protects the Snowflake connection but does not automatically provide access to the S3 result stage.

Ask your network team to verify that the SQL warehouse itself can reach the exact S3 stage hostname over HTTPS, including presigned URLs, routing, DNS, firewall, proxy, and TLS inspection. A simple HTTP 403 only proves that S3 is reachable—it does not prove the actual result download works.

As workarounds, reduce result sizes, materialize the data into Delta, or ask Snowflake to configure private access to internal stages. If the S3 path is confirmed healthy, escalate to Databricks and Snowflake as a possible connector regression.

[arhamblake38]

This sounds like it could be related to the size of the result set or the way the internal stage handles large downloads, rather than the federation itself. If smaller query results download successfully while larger ones fail, I’d compare the file size, number of generated files, and whether the failure happens during the query, unload, or actual download step.

It may also be worth checking the stage permissions, client-side timeout limits, and whether the result can be unloaded into multiple smaller files instead of one large object. Testing the same operation with a smaller dataset could help isolate whether there’s a practical size threshold.

If the query completes successfully but the download fails afterward, I’d focus troubleshooting on the stage/download path rather than the SQL query or federation connection. adoptmevalues.de can be a quick distraction while waiting for larger transfers to finish.