topic Re: Date schema issues with pyspark dataframe creation in Data Engineering

Date schema issues with pyspark dataframe creation

ckwan48 — Fri, 04 Nov 2022 03:34:25 GMT

I'm having some issues with creating a dataframe with a date column. Could I know what is wrong?

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType
from pyspark.sql.types import DateType, FloatType
 
spark = SparkSession.builder.appName('DataFrame').getOrCreate()
schema = StructType() \
      .add("DATE", DateType(), True) \
      .add("A", FloatType(), True) \
      .add("B", FloatType(), True)
 
df = spark.read.format("csv").option("header", True).option("dateFormat", "MM/dd/yyyy").schema(schem).load(''test.csv")
 
df.show()

This is the error I'm getting:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 158.0 failed 4 times, most recent failure: Lost task 0.3 in stage 158.0 (TID 1823) (10.237.208.145 executor 5): org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0:

Re: Date schema issues with pyspark dataframe creation

ckwan48 — Tue, 22 Nov 2022 17:22:05 GMT

Hi @Kaniz Fatma,

I actually changed the date format to 'M/d/Y' and it didn't throw any errors. I found in my csv file that it had dates like '3/1/2022'. Could that be the issue? But some dates also were like '12/1/2022. So I'm kind of confused.

Re: Date schema issues with pyspark dataframe creation

Debayan — Fri, 04 Nov 2022 05:06:32 GMT

Hi @Kevin Kim , Could you please try upgrading the spark version? Also, please provide the full error logs.