How to get the files one by one in blob storage using pyspark/python
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 08:21 AM
how to write the pyspark/python to get the files one by one in blob storage.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 11:55 AM
Hi @rameshybr ,
One way to do that is to get files from a given location using dbutils.fs.ls and then iterate over this list one file after another.
files = dbutils.fs.ls("abfss://conatiner@account_name.dfs.core.windows.net/folder")
for file in files:
df = spark.read.json(file.path)
#implement further logic
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 12:01 PM
# List files in a directory
files = dbutils.fs.ls("/mnt/<mount-name>/path/to/directory")
for file in files:
file_path = file.path
# Read each file into a DataFrame / if you file format is parquet for example i am taking
df = spark.read.format("parquet").load(file_path)
# Perform operations on the DataFrame
df.show()
df.write.mode("overwrite").parquet("if you want to write to any specific location")
Rishabh Pandey