topic Re: date type conversion error in Data Engineering

date type conversion error

ashkd7310 — Tue, 23 Jul 2024 02:55:39 GMT

Hello,

I am trying to convert the date in MM/dd/yyyy format. So I am first using the date_format function and converting the date into MM/dd/yyyy. So it becomes string. However, my use case is to have the data as date. so I am again converting the string into date by using to_date function. However, it becomes null when I am doing that way.

I have tried many ways but unable to solve. Is this something databricks doesn't support to store the date in the format of MM/dd/yyyy?

Re: date type conversion error

szymon_dybczak — Tue, 23 Jul 2024 04:51:14 GMT

Hi @ashkd7310 ,

It's not possible. Spark does not support date types formatted in some other way except for yyyy-MM-dd. If you need another format, you will need to again convert the date type into string type, but with the format which you need.

Re: date type conversion error

Rishabh-Pandey — Tue, 23 Jul 2024 07:27:32 GMT

Check with this method if it works.

# Convert date to MM/dd/yyyy format (string) df = df.withColumn("formatted_date", date_format("date", "MM/dd/yyyy")) # Convert string back to date df = df.withColumn("converted_date", to_date("formatted_date", "MM/dd/yyyy"))