topic databricks-connect, dbutils, abfss path, URISyntaxException in Data Engineering

databricks-connect, dbutils, abfss path, URISyntaxException

KrzysztofPrzyso — Tue, 06 Feb 2024 17:46:58 GMT

When trying to use `dbutils.fs.cp` in the #databricks-connect #databricks-connect context to upload files to Azure Datalake Gen2 I get a malformed URI error

I have used the code provided here:
https://learn.microsoft.com/en-gb/azure/databricks/dev-tools/databricks-connect/python/databricks-utilities

from databricks.sdk import WorkspaceClient
w = WorkspaceClient() 
path = r"abfss://bronze@devstorageacc.dfs.core.windows.net/test/"
w.dbutils.fs.cp('dbfs:/config.json', path)

Error:

```databricks.sdk.errors.mapping.InvalidParameterValue: java.net.URISyntaxException: Relative path in absolute URI: abfss:%5Cbronze@devstorageacc.dfs.core.windows.net%5Ctest```

The standard `dbutils.fs.cp` works on the cluster without problems. I have positively confirmed access rights.

Possibly it is a known issue described here: databricks-connect : Relative path in absolute URI · Issue #2883 · sparklyr/sparklyr (github.com)

Re: databricks-connect, dbutils, abfss path, URISyntaxException

Gen_Vi332 — Fri, 10 May 2024 01:42:23 GMT

I have the same problem.

Running :

from databricks.sdk import WorkspaceClient w = WorkspaceClient() w.dbutils.fs.ls("abfss://xxx@yyy.dfs.core.windows.net/zzz")

via Databricks Connect gives the same result as running :

from databricks.sdk import WorkspaceClient w = WorkspaceClient() w.dbutils.fs.ls("/")

The problem seems to be the presence of '//': in that case the string being passed to the databricks library seems to be just '/'

When doing the same directly on a normal Databricks Workspace Notebook it works:

dbutils.fs.ls("abfss://xxx@yyy.dfs.core.windows.net/zzz")

Re: databricks-connect, dbutils, abfss path, URISyntaxException

KrzysztofPrzyso — Mon, 10 Jun 2024 10:30:07 GMT

@Gen_Vi332 ,

Apologies for the late response,
My experiments with other solutions show that using Volume paths instead of `abfss://` paths can potentially solve the issue.
The drawback is that you need to expose the folder (or its parent) as Volume in Unity Catalog and grant the client access to the volume object.
`/Volumes/catalog_dev/operational/vlm_source_landing/file_path`
It seems that for many scenarios volume path is treated as local path, which can potentially simplify the code for non spark python operations.