raise Py4JJavaError while changing data type of a column

Marinagomes — Thu, 28 Mar 2024 16:01:02 GMT

I'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format

2/18/2021 7:20:12 PM

So I wrote following command

from pyspark.sql.functions import to_timestamp

(
spark.read
.table("convertToTimeStamp")
.withColumn("JoiningDate",to_timestamp("JoiningDate", "dd/MM/yyyy HH:mm:ss"))
.withColumn("EndDate",to_timestamp("EndDate", "dd/MM/yyyy HH:mm:ss"))
.write
.format("delta")
.mode("overwrite")
.option("overwriteSchema", True)
.saveAsTable("convertToTimeStamp")
)

But I'm getting error message saying that

raise Py4JJavaError(
"An error occurred while calling {0}{1}{2}.\n".
format(target_id, ".", name), value)

Can you please help me to resolve the issue?

topic raise Py4JJavaError while changing data type of a column in Data Engineering

raise Py4JJavaError while changing data type of a column