I think I ended up doing a truncate and mode = "append" so the structure stayed the same.

For truncate I had to get some direct access from the sparkcontext/sc variable

driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
connection = driver_manager.getConnection(mssql_url, mssql_user, mssql_pass)
connection.prepareCall("TRUNCATE TABLE my_table").execute()
connection.close()