databricks-connect, dbutils, abfss path, URISyntaxException

KrzysztofPrzyso
New Contributor III

‎02-06-2024 09:46 AM

When trying to use `dbutils.fs.cp` in the #databricks-connect #databricks-connect context to upload files to Azure Datalake Gen2 I get a malformed URI error

I have used the code provided here:
https://learn.microsoft.com/en-gb/azure/databricks/dev-tools/databricks-connect/python/databricks-ut...

 

from databricks.sdk import WorkspaceClient
w = WorkspaceClient() 
path = r"abfss://bronze@devstorageacc.dfs.core.windows.net/test/"
w.dbutils.fs.cp('dbfs:/config.json', path)

Error:

```databricks.sdk.errors.mapping.InvalidParameterValue: java.net.URISyntaxException: Relative path in absolute URI: abfss:%5Cbronze@devstorageacc.dfs.core.windows.net%5Ctest```

KrzysztofPrzyso_0-1707241094344.png

The standard `dbutils.fs.cp` works on the cluster without problems. I have positively confirmed access rights.

Possibly it is a known issue described here: databricks-connect : Relative path in absolute URI · Issue #2883 · sparklyr/sparklyr (github.com)

0 Kudos

Gen_Vi332
New Contributor II
In response to Retired_mod

‎05-09-2024 06:42 PM

I have the same problem.

Running :

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
w.dbutils.fs.ls("abfss://xxx@yyy.dfs.core.windows.net/zzz")

via Databricks Connect gives the same result as running :

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
w.dbutils.fs.ls("/")

The problem seems to be the presence of '//': in that case the string being passed to the databricks library seems to be just '/'

When doing the same directly on a normal Databricks Workspace Notebook it works:

dbutils.fs.ls("abfss://xxx@yyy.dfs.core.windows.net/zzz")
0 Kudos

KrzysztofPrzyso
New Contributor III
In response to Gen_Vi332

‎06-10-2024 03:30 AM

@Gen_Vi332 ,

Apologies for the late response,
My experiments with other solutions show that using Volume paths instead of `abfss://` paths can potentially solve the issue.
The drawback is that you need to expose the folder (or its parent) as Volume in Unity Catalog and grant the client access to the volume object. 
`/Volumes/catalog_dev/operational/vlm_source_landing/file_path`
It seems that for many scenarios volume path is treated as local path, which can potentially simplify the code for non spark python operations.

0 Kudos