https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83840#M37028 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/115833">@rameshybr</a>&nbsp;</P><LI-CODE lang="python"># List files in a directory files = dbutils.fs.ls("/mnt/&lt;mount-name&gt;/path/to/directory") for file in files: file_path = file.path # Read each file into a DataFrame / if you file format is parquet for example i am taking df = spark.read.format("parquet").load(file_path) # Perform operations on the DataFrame df.show() df.write.mode("overwrite").parquet("if you want to write to any specific location") </LI-CODE> Wed, 21 Aug 2024 19:01:20 GMT Rishabh-Pandey 2024-08-21T19:01:20Z https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83826#M37013 <P>how to write the pyspark/python to get the files one by one in blob storage.</P> Wed, 21 Aug 2024 15:21:28 GMT https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83826#M37013 rameshybr 2024-08-21T15:21:28Z https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83839#M37027 <P>Hi <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/115833">@rameshybr</a> ,</P><P>One way to do that is to get files from a given location using dbutils.fs.ls and then iterate over this list one file after another.<BR /><BR /></P><LI-CODE lang="markup">files = dbutils.fs.ls("abfss://conatiner@account_name.dfs.core.windows.net/folder") for file in files: df = spark.read.json(file.path) #implement further logic</LI-CODE> Wed, 21 Aug 2024 18:55:36 GMT https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83839#M37027 szymon_dybczak 2024-08-21T18:55:36Z https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83840#M37028 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/115833">@rameshybr</a>&nbsp;</P><LI-CODE lang="python"># List files in a directory files = dbutils.fs.ls("/mnt/&lt;mount-name&gt;/path/to/directory") for file in files: file_path = file.path # Read each file into a DataFrame / if you file format is parquet for example i am taking df = spark.read.format("parquet").load(file_path) # Perform operations on the DataFrame df.show() df.write.mode("overwrite").parquet("if you want to write to any specific location") </LI-CODE> Wed, 21 Aug 2024 19:01:20 GMT https://community.databricks.com/t5/data-engineering/how-to-get-the-files-one-by-one-in-blob-storage-using-pyspark/m-p/83840#M37028 Rishabh-Pandey 2024-08-21T19:01:20Z