Error truncating #REF with spark.read
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2024 12:58 PM
Hello guys
I am trying to read an excel file and even using PERMISSIVE mode, its truncating the records that contains #REF in any column 😥
Can anyone please help me on that?
schema = StructType([\
StructField('Col1', DateType(), True), \ <----------THIS COLUMN HAS #REF
StructField('Col2', DateType(), True), \
StructField('Col3', StringType(), True)])
test = (
spark.read.format("com.crealytics.spark.excel")
.option("header", header_option)
.option("parseMode", "PERMISSIVE")
.option("keepUndefinedRows", True)
.option("useNullForErrorCells", True)
.option("treatEmptyValuesAsNulls", True)
.option("setErrorCellsToFallbackValues", "true")
.option("maxRowsInMemory", 1000)
.option("useNullForErrorCells", True)
.option("dataAddress", "Test!A1:C200")
.schema(schema)
.load("File.xlsx")
