Unable to list a folder with square bracket in name

Loki466
New Contributor

‎01-18-2024 05:58 PM

I am trying to iterate over a container(adls gen2) in azure databricks using pyspark. basically I am using dbutils.fs.ls to list the contents for folder using recursive function.

this logic works perfectly fine for all folder except for the folders ehich has [] in its name.

I am getting java.uri exception saying that there are illegal characters, I tried to encode the folderpath but then I am getting filenotfound error as dbx is trying to list a folder that has encoded characters.

foldername: container/abd/qwe[rt]

Looks like this is a spark issue.

0 Kudos

NandiniN
Databricks Employee
Databricks Employee

‎12-11-2024 01:31 AM

[] comes under valid characters, so it is an issue with dbfs.fs.

dbutils.fs.ls("/Workspace/Users/user@databricks.com/qwe[rt]") fails with java.net.URISyntaxException: Illegal character in path at index

I tested a workaround meanwhile that can help you read the file.

%python
path = "file:/Workspace/Users/user@databricks.com/qwe\\[rt\\]/hello"
df = spark.read.text(path)
display(df)

For the Dbfs bug, I see we have an internal tracking ticket SC-163860.

Thanks!