topic Multiple sources found for csv in Data Engineering

Multiple sources found for csv

Jackson1111 — Mon, 22 Jul 2024 11:22:14 GMT

When I run a job using spark2 jar, I then run a python job to report :Multiple sources found for csv (org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2, org.apache.spark.sql.execution.datasources.csv.CSVFileFormat), please specify the fully qualified class name.

My python code is

df = self.spark.read.csv(full_data_file_path, header=False, schema=schema,sep=sep)

Re: Multiple sources found for csv

daniel_sahal — Mon, 22 Jul 2024 12:38:22 GMT

@Jackson1111
It looks like you've installed two different libraries to handle CSV data.
You need to specify which one you want to use, ex:

df = self.spark.read.format("org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2").option("header", False).option("schema", schema).option("sep", sep).load(full_data_file_path)