Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-11-2021 05:23 AM
According to the docs of
spark.read.csv(...) the path argument can be an RDD of strings:
path : str or list
string, or list of strings, for input path(s), or RDD of Strings storing CSV rows.
With that, you may use
spark.sparkContext.textFile(...) in combination with zipWithIndex(...) to perform the necessary row filtering. Putting things together this may look as follows:
n_skip_rows = ?
row_rdd = spark.sparkContext
.textFile(your_csv_file) \
.zipWithIndex() \
.filter(lambda row: row[1] >= n_skip_rows) \
.map(lambda row: row[0])
df = spark_session.read.csv(row_rdd, ...)
Hope that helps.