Hi All,
I need to easily find all the paths in a blob store to find the files and load them. I have tried using Azure Blob storage connection in python and I have a solution that works it is very slow. I was speaking to a data engineer, and he suggested I try using external connections to access the storage, but due to not being able to use hierarchical files system in the blob store I cannot use spark.read.load() to load all files as that requires HFS enabled and we can't on this storage.
The obvious candidate is to use os.walk, but that doesn't work, well not as you would use it for any regular filesystem. I tried using the abfss path as root path.
I could do it with the Azure Storage library and BlobServiceClient, but looking for any alternatives.
If anyone has a solution to geting all paths. if there is a solution that someone has worked out any tips would be great
def getdir_tree(root_path😞
path_list = []
path = os.path.join(root_path, 'targetdirectory')
for path, subdirs, files in os.walk(root_path):
for name in files:
path_list.append(os.path.join(path, name))
