BradSheridan
Databricks Partner

Answered my own question!! check this out:

dropSQL = ("DROP TABLE IF EXISTS <tablename>;"). --note the semicolon at the end!

createSQL = ("CREATE TABLE IF NOT EXISTS <tablename> (field1 int, field2 date, etc...);")

preActionsSQL = dropSQL + createSQL

then when writing your dataframe to a Redshift table, include this:

.option("preactions", preActionsSQL)

Side note: if you use the .mode("overwrite") option, I *think it actually either drops or truncates the target table first anyway, but I haven't confirmed this...was too excited to share my findings with y'all !!!!

View solution in original post