From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the PUT and returns a signed upload URL, but the client upload to the storage gateway fails.
I hit this via the open-source dlt (data load tool) Databricks destination, whose "direct load" mode stages files with this exact PUT. I've reproduced it with the bare connector and no dlt, in two separate workspaces.
Environment
- Cloud/region: Azure, East US (sn=az-eastus-c3 in the signed URL)
- Compute: serverless notebook, environment version [...]
- databricks-sql-connector 4.8.0
- dlt [...] (not needed for the repro)
- Warehouse: serverless SQL warehouse
- Metastore: East US Unity Catalog metastore
Network / egress configuration (checked in the account console)
- Network connectivity configuration (NCC): None
- Network policy: default-policy, egress "Allow access to all destinations"
- Compliance security profile: Disabled
- system.access.outbound_network: no rows at all for the account, so no denied events are logged
Error
HTTPSConnectionPool(host='eastus.storage.azuredatabricks.net', port=443): Max retries exceeded with url:
/api/2.0/fs/files/Volumes/CATALOG/SCHEMA/VOLUME/put_test.txt?...&X-Databricks-Consumer-Network-Zone=internet&sn=az-eastus-c3&...
(Caused by NewConnectionError(... [Errno 111] Connection refused))
What I found
- Inside serverless, eastus.storage.azuredatabricks.net resolves to 192.168.200.28 (private), which refuses port 443.
- Outside Databricks, the same hostname resolves to 52.234.168.242 (public).
- From the same serverless compute, forcing the connection to 52.234.168.242 completes TLS and returns HTTP 400 (expected for an unsigned request), so the gateway itself is reachable and only the internal DNS answer is wrong.
- The signed URL carries X-Databricks-Consumer-Network-Zone=internet, yet serverless routes that host to an internal IP.
- From the same serverless compute, writing to the same volume works through:
- /Volumes/... FUSE (shutil.copy): โ
- Databricks SDK WorkspaceClient().files.upload() (workspace host): โ
- SQL connector PUT (storage gateway): โ
- Reproduced in a brand-new serverless workspace with default settings: the same private IP, the same error, the same az-eastus-c3.
- The same PUT worked on classic compute in the original workspace.
Minimal repro (serverless notebook)
In serverless notebooks, the built-in databricks package shadows the connector's databricks.sql, so the path patch is needed:
%pip install databricks-sql-connector
dbutils.library.restartPython()
import sys, os, socket, databricks
for p in sys.path:
c = os.path.join(p, "databricks")
if os.path.isdir(os.path.join(c, "sql")) and c not in databricks.__path__:
databricks.__path__.append(c)
from databricks import sql
print(socket.gethostbyname("eastus.storage.azuredatabricks.net")) # -> 192.168.200.28
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
open("/tmp/t.txt", "w").write("hi")
with sql.connect(server_hostname="<workspace-host>",
http_path="/sql/1.0/warehouses/<id>",
access_token=ctx.apiToken().get(),
staging_allowed_local_path="/tmp") as conn:
conn.cursor().execute(
"PUT '/tmp/t.txt' INTO '/Volumes/<catalog>/<schema>/<volume>/t.txt' OVERWRITE")Questions
- Is SQL connector PUT from inside serverless compute supported? The volume docs list PUT under external tools, and the serverless limitations page doesn't mention it either way.
- If it is supported, is the private DNS answer for eastus.storage.azuredatabricks.net on serverless a known issue (East US / az-eastus-c3)?
- Has anyone seen this in other regions?
