Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2023 12:51 AM
Ok some tips:
- you do not have to create a spark session on databricks, it is already created by databricks. But your getOrCreate does not break anything
spark can read whole folders at once. If you have 12 csv files in one folder, AND THEY HAVE THE SAME SCHEMA, you can try: spark.read.format("csv").option("delimiter", ",").option("header","true").load("dbfs:/FileStore/aleksandra.frolova@zebra.com/Sales Analysis/)
Like that the whole folder is read.
Of course, if your files have a different structure, spark does not know what to do obviously so you will have to define a schema manually.