Registered to post this so forgive the formatting nightmare

This is a python databricks script function that allows you to convert from string to datetime or date and utilising coalesce

def to_datetime_(col, format): # Spark 2.2 or later syntax, for < 2.2 use unix_timestamp and cast return coalesce(to_date(col, format))

df3 = sqlContext.sql(sqlString2)df3.withColumn(df3['CalDay'],pd.to_datetime_(df3['CalDay'],format = '%Y%m%d'))