- 38404 Views
- 8 replies
- 2 kudos
Skip number of rows when reading CSV files
staticDataFrame = spark.read.format("csv")\ .option("header", "true").option("inferSchema", "true").load("/FileStore/tables/Consumption_2019/*.csv") when above, I need an option to skip say first 4 lines on each CSV file, How do I do that?
- 38404 Views
- 8 replies
- 2 kudos
Latest Reply
- 2 kudos
The option... .option("skipRows", <number of rows to skip>) ...works for me as well. However, I am surprised that the official Spark doc does not list it as a CSV Data Source Option: https://spark.apache.org/docs/latest/sql-data-sources-csv.html#data...
- 2 kudos