How to add a new datetime column to a spark dataFrame from existing timestamp column

User16790091296
Databricks Employee
Databricks Employee

‎06-24-2021 08:07 AM

I have a data frame in Spark that has a column timestamp. I want to add a new column to this data frame that has the DateTime in the below format created from this existing timestamp column.

“YYYY-MM-DD HH:MM:SS”

Labels:
0 Kudos

Srikanth_Gupta_
Databricks Employee
Databricks Employee

‎06-25-2021 06:07 AM

val df = Seq(("2021-11-05 02:46:47.154410"),("2019-10-05 2:46:47.154410")).toDF("old_column")

display(df)

import org.apache.spark.sql.functions._

val df2 = df.withColumn("new_column", from_unixtime(unix_timestamp(col("old_column"), "yyyy-MM-dd HH:mm:ss.SSSSSS"),"yyyy-MM-dd HH:mm:ss"))

display(df2)

I have tested this and this should work

0 Kudos