How to add a new datetime column to a spark dataFrame from existing timestamp column

User16790091296 — Thu, 24 Jun 2021 15:07:43 GMT

I have a data frame in Spark that has a column timestamp. I want to add a new column to this data frame that has the DateTime in the below format created from this existing timestamp column.

“YYYY-MM-DD HH:MM:SS”

Re: How to add a new datetime column to a spark dataFrame from existing timestamp column

Srikanth_Gupta_ — Fri, 25 Jun 2021 13:07:59 GMT

val df = Seq(("2021-11-05 02:46:47.154410"),("2019-10-05 2:46:47.154410")).toDF("old_column")

display(df)

import org.apache.spark.sql.functions._

val df2 = df.withColumn("new_column", from_unixtime(unix_timestamp(col("old_column"), "yyyy-MM-dd HH:mm:ss.SSSSSS"),"yyyy-MM-dd HH:mm:ss"))

display(df2)

I have tested this and this should work

topic Re: How to add a new datetime column to a spark dataFrame from existing timestamp column in Data Engineering

How to add a new datetime column to a spark dataFrame from existing timestamp column

Re: How to add a new datetime column to a spark dataFrame from existing timestamp column