I'll share I'm having a variant of the same issue. I have a varchar field in the form YYYYMMDD which I'm trying to join to another varchar field from another table in the form of MM/DD/YYYY. Does anyone know of a way to do this in SPARK SQL without splitting the YEAR, MONTH and Day and concatenating them together? I've tried date_format, to_date, date, cast, and to_timestamp functions or different combinations of them, but haven't yet succeeded. Any help is appreciated. Thanks!