- 27779 Views
- 10 replies
- 5 kudos
Not able to read text file from local file path - Spark CSV reader
We are using Spark CSV reader to read the csv file to convert as DataFrame and we are running the job on yarn-client, its working fine in local mode. We are submitting the spark job in edge node. But when we place the file in local file path instead...
- 27779 Views
- 10 replies
- 5 kudos
- 6076 Views
- 2 replies
- 1 kudos
Handle comma inside cell of CSV
We are using spark-csv_2.10 > version 1.5.0 and reading the csv file column which contains comma " , " as one of the character. The problem we are facing is like that it treats the rest of line after the comma as new column and data is not interpre...
- 6076 Views
- 2 replies
- 1 kudos
- 1 kudos
Take a look here for options, http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=dataframereader#pyspark.sql.DataFrameReader.csv If a csv file has commas then the tradition is to quote the string that contains the comma, In ...
- 1 kudos