Hi,

Thank you for your reply. I have been able to read all the files in a single attempt using pyspark:

spk_df1 = spark.read.text(all_filepaths.tolist()).option("header", "true").format("csv").option(delimiter = ',')

However, I could not figure out how to perform any of the row-level transformations that are unique to each file. What would you suggest here?