@Jackson1111 
It looks like you've installed two different libraries to handle CSV data.
You need to specify which one you want to use, ex:

df = self.spark.read.format("org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2").option("header", False).option("schema", schema).option("sep", sep).load(full_data_file_path)