How to get the files one by one in blob storage using pyspark/python

[rameshybr]

how to write the pyspark/python to get the files one by one in blob storage.

[szymon_dybczak]

Hi @rameshybr ,

One way to do that is to get files from a given location using dbutils.fs.ls and then iterate over this list one file after another.

files = dbutils.fs.ls("abfss://conatiner@account_name.dfs.core.windows.net/folder")

for file in files:
    df = spark.read.json(file.path)
    #implement further logic

[Rishabh-Pandey]

@rameshybr 

# List files in a directory
files = dbutils.fs.ls("/mnt/<mount-name>/path/to/directory")

for file in files:
    file_path = file.path
    # Read each file into a DataFrame / if you file format is parquet for example i am taking
    df = spark.read.format("parquet").load(file_path)  
    
    # Perform operations on the DataFrame
    df.show()
    
    df.write.mode("overwrite").parquet("if you want to write to any specific location")