Hello @sudopradip, I took a look at both internal and external documentation and here is what I found.
Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, donโt pin the public IP, and open a support ticket. @DATA_pulseโs AWS us-east-2 reproduction turned this from an East US oddity into a platform-wide pattern. And your own narrowing to catalogs on Databricks default storage is the piece that ties it together.
Whatโs happening. A connector PUT runs in two steps: the warehouse accepts the statement and returns a short-lived signed URL, then the Python process uploads the file to that URL from wherever itโs running. For a volume on default storage, that URL points at the regional default-storage gateway (eastus.storage.azuredatabricks.net on Azure, us-east-2.storage.cloud.databricks.com on AWS), the gateway the IP and domain docs cover for default storage. For a catalog on your own storage account, the signed URL points at your ADLS account and the gateway never comes into play, which is why the stock dlt load worked in your hybrid workspace. Inside serverless, the gateway hostname resolves to 192.168.200.28 and nothing answers on 443. The identical private address on two clouds suggests the serverless network plane steers that hostname internally on purpose, and whatever sits there doesnโt accept an upload signed for the internet zone (X-Databricks-Consumer-Network-Zone=internet). The /Volumes path, files.upload() through your workspace host, classic compute, and a laptop all succeed because none of them depend on that DNS answer.
On your questions. The docs list PUT INTO, GET, and REMOVE as connector-only commands for external tools, and the default storage page says external clients reach volumes on default storage through the Files API and tables through ODBC and JDBC. Nothing says yes or no about the connectorโs PUT running inside serverless against default storage, and the serverless limitations page is silent too, so Iโd treat it as unsupported until that changes. I donโt know of a fix or a public known-issue entry, and the default storage page was updated this week without one. Fresh default workspaces on two clouds showing the same private IP points at platform behavior, not your configuration. @data_pulse has answered the regions question for AWS us-east-2, with connector 4.4.0 and 4.8.0 both failing.
Workarounds. Inside a notebook youโre already in the workspace, so the connector isnโt needed to reach a volume:
python
import shutil
shutil.copy("/tmp/file.csv", "/Volumes/<catalog>/<schema>/<volume>/file.csv")
python
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
with open("/tmp/file.csv", "rb") as f:
w.files.upload("/Volumes/<catalog>/<schema>/<volume>/file.csv", f, overwrite=True)
Keep PUT INTO for code that runs outside serverless. For dlt, your DatabricksLoadJob subclass overriding _handle_local_file_upload to call files.upload() is the cleanest no-infrastructure fix today, and your dlt issue #4606 proposing a volume_upload_method option is still open as of this week. @SumeshKashyapโs alternatives stand: host the dlt process on classic or job compute, or stage to an ADLS Gen2 external location and load with COPY INTO, which skips the gateway upload entirely. Donโt hardcode the public IP or override DNS in /etc/hosts; those addresses change and youโd be bypassing the routing serverless intends. Allowlisting the 192.168.x.x address wonโt help either, since the refusal happens at a Databricks-managed endpoint before anything reaches storage.
Getting it fixed. Open a support ticket so itโs tracked, and include:
- both workspace IDs, the region and the
sn= value
- the serverless environment version and the connector versions tested
- the private DNS answer inside serverless versus the public one elsewhere
- the minimal repro, plus confirmation that
/Volumes, files.upload(), and classic compute all work
- that the catalog is on default storage and the same load works on a catalog in your own storage account
- your network policy and NCC state, and a link to @DATA_pulseโs AWS result
Either outcome helps: a fix, or a line in the limitations docs so the next person doesnโt lose a weekend to it. If you hear back, please post the result here.
References:
Regards,
Louis
