Load multiple csv files into a dataframe in order

Shridhar
New Contributor

‎10-17-2018 06:24 PM

I can load multiple csv files by doing something like:

paths = ["file_1", "file_2", "file_3"]
df = sqlContext.read
       .format("com.databricks.spark.csv")
       .option("header", "true")
       .load(paths)

But this doesn't seem to preserve the order in |paths|.

In particular, I'm trying to have a monotonically increasing id that spans the data in all files.

Labels:
0 Kudos

JayaKommuru
New Contributor II

‎11-19-2019 07:50 PM

@shridhar have you found out an alternative for achieving this. I also have the same problem.

Jaswanth_Saniko
New Contributor III

‎01-12-2022 04:43 AM 

val diamonds = spark.read.format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("/FileStore/tables/11.csv","/FileStore/tables/12.csv","/FileStore/tables/13.csv")
 
display(diamonds)

This is working for me @Shridhar​ 

View solution in original post