Load multiple csv files into a dataframe in order

[Shridhar]

I can load multiple csv files by doing something like:

paths = ["file_1", "file_2", "file_3"]
df = sqlContext.read
       .format("com.databricks.spark.csv")
       .option("header", "true")
       .load(paths)

But this doesn't seem to preserve the order in |paths|.

In particular, I'm trying to have a monotonically increasing id that spans the data in all files.

[JayaKommuru]

@shridhar have you found out an alternative for achieving this. I also have the same problem.

[Jaswanth_Saniko · Accepted solution]

val diamonds = spark.read.format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("/FileStore/tables/11.csv","/FileStore/tables/12.csv","/FileStore/tables/13.csv")
 
display(diamonds)

This is working for me @Shridhar​