topic Re: Unable to list a folder with square bracket in name in Get Started Discussions

Unable to list a folder with square bracket in name

Loki466 — Fri, 19 Jan 2024 01:58:01 GMT

I am trying to iterate over a container(adls gen2) in azure databricks using pyspark. basically I am using dbutils.fs.ls to list the contents for folder using recursive function.

this logic works perfectly fine for all folder except for the folders ehich has [] in its name.

I am getting java.uri exception saying that there are illegal characters, I tried to encode the folderpath but then I am getting filenotfound error as dbx is trying to list a folder that has encoded characters.

foldername: container/abd/qwe[rt]

Looks like this is a spark issue.

Re: Unable to list a folder with square bracket in name

NandiniN — Wed, 11 Dec 2024 09:31:47 GMT

[] comes under valid characters, so it is an issue with dbfs.fs.

dbutils.fs.ls("/Workspace/Users/user@databricks.com/qwe[rt]") fails with java.net.URISyntaxException: Illegal character in path at index

I tested a workaround meanwhile that can help you read the file.

%python

path = "file:/Workspace/Users/user@databricks.com/qwe\\[rt\\]/hello"

df = spark.read.text(path)

display(df)

For the Dbfs bug, I see we have an internal tracking ticket SC-163860.

Thanks!