Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2023 08:18 AM
df = spark.read.csv("/mnt/lake/data/csv")Here I assume "/mnt/lake/data/csv" is the directory with the 5 files.
spark.read.csv also has some options like the separator, header etc:
https://spark.apache.org/docs/latest/sql-data-sources-csv.html
So there is no need to do this one by one, read the whole dir in one go.