What I am doing:

spark_df = spark.createDataFrame(dfnew)

spark_df.write.saveAsTable("default.test_table", index=False, header=True)

This automatically detects the datatypes and is working right now. BUT, what if the datatype cannot be detected or detects wrong? Mostly concerned about doubles, ints, bigints.

I tested casting but it doesnt work on databricks:

spark_df = spark.createDataFrame(dfnew.select(dfnew("Year").cast(IntegerType).as("Year")))

Is there a way to feed a DDL to spark dataframe for databricks? Should I not use spark to create the table?