Hubert-Dudek
Databricks MVP

New Your Taxi data from your example is already included in your workspace as it is demo dataset.

It is enough to read "yellow" folder and it will read all csvs from there.

If you want to save it as a single file you can do .repartition(1).write.csv(destination_folder).save()

image.png


My blog: https://databrickster.medium.com/