topic Change datetime format from one to another without changing datatype in databricks sql in Data Engineering

Change datetime format from one to another without changing datatype in databricks sql

richakamat130 — Thu, 26 Sep 2024 08:10:52 GMT

Change datetime"2002-01-01T00:00:00.000" to 'MM/dd/yyyy HH:mm:ss' format without changing datatype/ having it in datetime data type

Re: Change datetime format from one to another without changing datatype in databricks sql

Mister-Dinky — Thu, 26 Sep 2024 12:31:21 GMT

I presume then that the "datetime" string you provided is a string column?
If so, try the following code. This will not perform any transformation to timestamps:

from pyspark.sql.functions import lit, split, concat data = [{"date_string": "2002-01-01T00:00:00.000"}] data_df = spark.createDataFrame(data) \ .withColumn( "date_string_reformatted", concat( lit(split("date_string", "-")[1]), lit("-"), lit(split(split("date_string", "-")[2], "T")[0]), lit("-"), lit(split("date_string", "-")[0]), lit(" "), lit(split(split(split("date_string", "-")[2], "T")[1], "\.")[0]), ) ) data_df.display()

The result:

Re: Change datetime format from one to another without changing datatype in databricks sql

szymon_dybczak — Thu, 26 Sep 2024 13:03:49 GMT

Hi @Mister-Dinky ,

I think what @richakamat130 would like to obtain is to format datetime according to his needs, but to keep orginal data format (in this case datetime). Unfortunately, this is not possible in spark. If you want to have different display format then you need to have this conversion.

Re: Change datetime format from one to another without changing datatype in databricks sql

Mister-Dinky — Thu, 26 Sep 2024 13:10:08 GMT

Ah, the "without changing datatype/ having it in datetime data type" can be interpreted in that way as well. Then I agree with your statement.

Re: Change datetime format from one to another without changing datatype in databricks sql

filipniziol — Thu, 26 Sep 2024 13:53:24 GMT

Hi @Mister-Dinky ,

As @szymon_dybczak if you have a datetime, then you have a datetime.

What you see is just a format of defined in the Databricks UI. Other applications may display it differently depending on the defaults, regional formats etc.

If you want to see datetime displayed differently in Databricks UI, then just navigate to your user settings and change it accordingly to your preferences: