How to add a new datetime column to a spark dataFrame from existing timestamp column

[User16790091296]

I have a data frame in Spark that has a column timestamp. I want to add a new column to this data frame that has the DateTime in the below format created from this existing timestamp column.

“YYYY-MM-DD HH:MM:SS”

[Srikanth_Gupta_]

val df = Seq(("2021-11-05 02:46:47.154410"),("2019-10-05 2:46:47.154410")).toDF("old_column")

display(df)

import org.apache.spark.sql.functions._

val df2 = df.withColumn("new_column", from_unixtime(unix_timestamp(col("old_column"), "yyyy-MM-dd HH:mm:ss.SSSSSS"),"yyyy-MM-dd HH:mm:ss"))

display(df2)

I have tested this and this should work

[Kaniz]

You can utilise to_timestamp

[Kaniz]

from pyspark import SparkContext

from pyspark.sql import SQLContext

from functools import reduce

import pyspark.sql.functions as F

sc = SparkContext.getOrCreate()

sql = SQLContext(sc)

input_list = [

 (1,"2019-11-07 10:30:00")  ,(1,"2019-11-08 10:30:00")

  ,(1,"2019-11-09 10:30:00")

  ,(1,"2019-11-11 10:30:00")

  ,(1,"2019-11-12 10:30:00")

  ,(1,"2019-11-13 10:30:00")

  ,(1,"2019-11-14 10:30:00")

  ,(2,"2019-11-08 10:30:00")

  ,(2,"2019-11-09 10:30:00")

  ,(3,"2019-11-09 10:30:00")

  ,(3,"2019-11-10 10:30:00")

  ,(3,"2019-11-11 10:30:00")

  ,(2,"2019-11-15 10:30:00")

  ,(2,"2019-11-18 10:30:00")

  ,(4,"2019-11-10 10:30:00")

  ,(4,"2019-11-11 10:30:00") ]

sparkDF = sql.createDataFrame(input_list,['customerid','date'])

sparkDF = sparkDF.withColumn('date_timestamp',F.to_timestamp(F.col('date'), 'yyyy-MM-dd HH:mm:ss')) sparkDF.show()