Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 12:01 PM
# List files in a directory
files = dbutils.fs.ls("/mnt/<mount-name>/path/to/directory")
for file in files:
file_path = file.path
# Read each file into a DataFrame / if you file format is parquet for example i am taking
df = spark.read.format("parquet").load(file_path)
# Perform operations on the DataFrame
df.show()
df.write.mode("overwrite").parquet("if you want to write to any specific location")
Rishabh Pandey