topic Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve in Data Engineering

Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves

sudopradip — Mon, 26 Oct 2026 10:15:10 GMT

From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the PUT and returns a signed upload URL, but the client upload to the storage gateway fails.

I hit this via the open-source dlt (data load tool) Databricks destination, whose "direct load" mode stages files with this exact PUT. I've reproduced it with the bare connector and no dlt, in two separate workspaces.

Environment

Cloud/region: Azure, East US (sn=az-eastus-c3 in the signed URL)
Compute: serverless notebook, environment version [...]
databricks-sql-connector 4.8.0
dlt [...] (not needed for the repro)
Warehouse: serverless SQL warehouse
Metastore: East US Unity Catalog metastore

Network / egress configuration (checked in the account console)

Network connectivity configuration (NCC): None
Network policy: default-policy, egress "Allow access to all destinations"
Compliance security profile: Disabled
system.access.outbound_network: no rows at all for the account, so no denied events are logged

Error

HTTPSConnectionPool(host='eastus.storage.azuredatabricks.net', port=443): Max retries exceeded with url:
/api/2.0/fs/files/Volumes/CATALOG/SCHEMA/VOLUME/put_test.txt?...&X-Databricks-Consumer-Network-Zone=internet&sn=az-eastus-c3&...
(Caused by NewConnectionError(... [Errno 111] Connection refused))

What I found

Inside serverless, eastus.storage.azuredatabricks.net resolves to 192.168.200.28 (private), which refuses port 443.
Outside Databricks, the same hostname resolves to 52.234.168.242 (public).
From the same serverless compute, forcing the connection to 52.234.168.242 completes TLS and returns HTTP 400 (expected for an unsigned request), so the gateway itself is reachable and only the internal DNS answer is wrong.
The signed URL carries X-Databricks-Consumer-Network-Zone=internet, yet serverless routes that host to an internal IP.
From the same serverless compute, writing to the same volume works through:
- /Volumes/... FUSE (shutil.copy): ✅
- Databricks SDK WorkspaceClient().files.upload() (workspace host): ✅
- SQL connector PUT (storage gateway): ❌
Reproduced in a brand-new serverless workspace with default settings: the same private IP, the same error, the same az-eastus-c3.
The same PUT worked on classic compute in the original workspace.

Minimal repro (serverless notebook)

In serverless notebooks, the built-in databricks package shadows the connector's databricks.sql, so the path patch is needed:

%pip install databricks-sql-connector

dbutils.library.restartPython()

import sys, os, socket, databricks
for p in sys.path:
    c = os.path.join(p, "databricks")
    if os.path.isdir(os.path.join(c, "sql")) and c not in databricks.__path__:
        databricks.__path__.append(c)
from databricks import sql

print(socket.gethostbyname("eastus.storage.azuredatabricks.net"))  # -> 192.168.200.28

ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
open("/tmp/t.txt", "w").write("hi")
with sql.connect(server_hostname="<workspace-host>",
                 http_path="/sql/1.0/warehouses/<id>",
                 access_token=ctx.apiToken().get(),
                 staging_allowed_local_path="/tmp") as conn:
    conn.cursor().execute(
        "PUT '/tmp/t.txt' INTO '/Volumes/<catalog>/<schema>/<volume>/t.txt' OVERWRITE")

Questions

Is SQL connector PUT from inside serverless compute supported? The volume docs list PUT under external tools, and the serverless limitations page doesn't mention it either way.
If it is supported, is the private DNS answer for eastus.storage.azuredatabricks.net on serverless a known issue (East US / az-eastus-c3)?
Has anyone seen this in other regions?

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

SumeshKashyap — Mon, 26 Oct 2026 11:52:24 GMT

Hi @sudopradip

Short version: the SQL connector's PUT/GET is a staging path meant for external clients. From inside serverless, the storage-gateway hostname resolves through the serverless plane's private DNS to an endpoint that doesn't serve that traffic, so you can't fix it from your code. Inside Databricks, write to the volume through FUSE or the Files API (both already work for you), and open a support ticket for the DNS behaviour.

On your questions:

1. Supported from inside serverless? I couldn't find any doc that says it is. The volumes docs list PUT/GET INTO under connecting from external tools and drivers. For code running on Databricks compute, the documented options are the /Volumes FUSE path and the Files API (WorkspaceClient().files.upload). Your classic-compute success fits this: classic resolves the gateway through normal Azure DNS, while serverless resolves it inside the Databricks-managed network.

2. Private IP for eastus.storage.azuredatabricks.net: you've shown the gateway is reachable on its public IP and only the in-plane DNS answer is wrong, and it happens in a fresh default workspace with no NCC. That puts it on the platform side. Raise it with Azure Databricks support and include the workspace IDs, region/sn=az-eastus-c3, the 192.168.200.28 vs 52.234.168.242 resolution, and your minimal repro. Don't hardcode the public IP as a workaround: those IPs can change, and it bypasses whatever routing serverless intends.

3. Other regions: I haven't tested this outside East US.

Workarounds for the dlt load in the meantime:

- Run the dlt pipeline from classic (job) compute, where direct load works as you've seen.

- Or configure dlt staging to an ADLS Gen2 container that's a UC external location (filesystem staging with an az:// bucket_url plus staging credentials). The databricks destination then loads with COPY INTO from that location and never calls the PUT gateway.

- If you control the load code, write files with shutil.copy to /Volumes/... or files.upload(), then run COPY INTO / read_files from the volume path.

Docs: https://learn.microsoft.com/azure/databricks/volumes/volume-files and https://dlthub.com/docs/dlt-ecosystem/destinations/databricks

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

SumeshKashyap — Mon, 26 Oct 2026 14:06:21 GMT

hello @sudopradip , if the response was helpful, please accept the reply as answer so that it is beneficial for other community members

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

sudopradip — Mon, 26 Oct 2026 14:35:53 GMT

Update: narrowed scope. It fails when the catalog is on Databricks-managed storage (storage Databricks owns, outside our Azure subscription, which a serverless workspace uses for a catalog created with no location). In a hybrid workspace, the catalog Databricks creates automatically sits on the workspace's own storage account in our Azure subscription; with that catalog, the same stock dlt load from serverless compute works.

thanks @SumeshKashyap
that make sense
i think i will keep issue open until it's officially documented or confirmed

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

data_pulse — Mon, 26 Oct 2026 15:45:27 GMT

I tested this independently and was able to reproduce a very similar failure pattern on AWS us-east-2, using a Databricks serverless.

Here are the results against the same UC Volume:

Upload method Result

Direct /Volumes/.. write	PASS (byte readback verified)
Databricks SDK Files API (0.40.0)	PASS (byte readback verified)
SQL Connector 4.8.0 PUT INTO	FAIL (twice)
SQL Connector 4.4.0 PUT INTO	FAIL (twice)
SQL Connector 4.8.0 from local Windows	PASS (twice, byte readback verified)

The interesting part is the gateway connectivity:

Inside the serverless notebook, us-east-2.storage.cloud.databricks.com resolved to 192.168.200.28, with TCP port 443 returning [Errno 111] Connection refused.
Warehouse authentication and simple SELECT succeeded, so the failure occurred later, during the connector's storage-gateway upload.
From local Windows, the same gateway resolved to public IP addresses, TCP 443 succeeded, and uploads to the same warehouse and Volume worked

This confirms that similar symptoms beyond Azure East US, including on AWS us-east-2.

It also suggests the behaviour is not limited to SQL Connector 4.8.0.

The results point toward a storage gateway connectivity issue specific to the serverless execution environment, rather than general UC Volume permissions or SQL warehouse connectivity.

Hopefully this cross-cloud reproduction helps narrow down the investigation.

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

sudopradip — Thu, 29 Oct 2026 06:13:01 GMT

Thanks @data_pulse 🙌 This helps a lot! We can now rule out my Databricks setup as the cause; it's a real but.

The fact that it works from Windows points to how the core service handles Databricks-managed storage access from notebooks, narrow down the issue.

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

data_pulse — Thu, 29 Oct 2026 08:41:55 GMT

Welcome @sudopradip . Glad it helped to narrow down further.

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

Louis_Frolio — Sat, 07 Nov 2026 17:45:37 GMT

Hello @sudopradip, I took a look at both internal and external documentation and here is what I found.

Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, don’t pin the public IP, and open a support ticket. @DATA_pulse’s AWS us-east-2 reproduction turned this from an East US oddity into a platform-wide pattern. And your own narrowing to catalogs on Databricks default storage is the piece that ties it together.

What’s happening. A connector PUT runs in two steps: the warehouse accepts the statement and returns a short-lived signed URL, then the Python process uploads the file to that URL from wherever it’s running. For a volume on default storage, that URL points at the regional default-storage gateway (eastus.storage.azuredatabricks.net on Azure, us-east-2.storage.cloud.databricks.com on AWS), the gateway the IP and domain docs cover for default storage. For a catalog on your own storage account, the signed URL points at your ADLS account and the gateway never comes into play, which is why the stock dlt load worked in your hybrid workspace. Inside serverless, the gateway hostname resolves to 192.168.200.28 and nothing answers on 443. The identical private address on two clouds suggests the serverless network plane steers that hostname internally on purpose, and whatever sits there doesn’t accept an upload signed for the internet zone (X-Databricks-Consumer-Network-Zone=internet). The /Volumes path, files.upload() through your workspace host, classic compute, and a laptop all succeed because none of them depend on that DNS answer.

On your questions. The docs list PUT INTO, GET, and REMOVE as connector-only commands for external tools, and the default storage page says external clients reach volumes on default storage through the Files API and tables through ODBC and JDBC. Nothing says yes or no about the connector’s PUT running inside serverless against default storage, and the serverless limitations page is silent too, so I’d treat it as unsupported until that changes. I don’t know of a fix or a public known-issue entry, and the default storage page was updated this week without one. Fresh default workspaces on two clouds showing the same private IP points at platform behavior, not your configuration. @data_pulse has answered the regions question for AWS us-east-2, with connector 4.4.0 and 4.8.0 both failing.

Workarounds. Inside a notebook you’re already in the workspace, so the connector isn’t needed to reach a volume:

python

import shutil
shutil.copy("/tmp/file.csv", "/Volumes/<catalog>/<schema>/<volume>/file.csv")

python

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
with open("/tmp/file.csv", "rb") as f:
    w.files.upload("/Volumes/<catalog>/<schema>/<volume>/file.csv", f, overwrite=True)

Keep PUT INTO for code that runs outside serverless. For dlt, your DatabricksLoadJob subclass overriding _handle_local_file_upload to call files.upload() is the cleanest no-infrastructure fix today, and your dlt issue #4606 proposing a volume_upload_method option is still open as of this week. @SumeshKashyap’s alternatives stand: host the dlt process on classic or job compute, or stage to an ADLS Gen2 external location and load with COPY INTO, which skips the gateway upload entirely. Don’t hardcode the public IP or override DNS in /etc/hosts; those addresses change and you’d be bypassing the routing serverless intends. Allowlisting the 192.168.x.x address won’t help either, since the refusal happens at a Databricks-managed endpoint before anything reaches storage.

Getting it fixed. Open a support ticket so it’s tracked, and include:

both workspace IDs, the region and the sn= value
the serverless environment version and the connector versions tested
the private DNS answer inside serverless versus the public one elsewhere
the minimal repro, plus confirmation that /Volumes, files.upload(), and classic compute all work
that the catalog is on default storage and the same load works on a catalog in your own storage account
your network policy and NCC state, and a link to @DATA_pulse’s AWS result

Either outcome helps: a fix, or a line in the limitations docs so the next person doesn’t lose a weekend to it. If you hear back, please post the result here.

References:

PUT INTO (connector-only): https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-aux-connector-put-into
Default storage in Databricks: https://learn.microsoft.com/azure/databricks/storage/default-storage
IP addresses and domains, including default storage gateways: https://learn.microsoft.com/azure/databricks/resources/ip-domain-region (Azure), https://docs.databricks.com/aws/en/resources/ip-domain-region (AWS)
Work with files in Unity Catalog volumes: https://learn.microsoft.com/azure/databricks/volumes/volume-files
Serverless compute limitations: https://learn.microsoft.com/azure/databricks/compute/serverless/limitations
Databricks SDK for Python: https://learn.microsoft.com/azure/databricks/dev-tools/sdk-python
Databricks SQL Connector for Python: https://docs.databricks.com/aws/en/dev-tools/python-sql-connector
dlt issue #4606: https://github.com/dlt-hub/dlt/issues/4606

Regards,
Louis