I am trying to iterate over a container(adls gen2) in azure databricks using pyspark. basically I am using dbutils.fs.ls to list the contents for folder using recursive function.

this logic works perfectly fine for all folder except for the folders ehich has [] in its name.

I am getting java.uri exception saying that there are illegal characters, I tried to encode the folderpath but then I am getting filenotfound error as dbx is trying to list a folder that has encoded characters.

foldername: container/abd/qwe[rt]

Looks like this is a spark issue.