Yeah in my case I'm starting to think this could be an installation thing, or a unset property. So I created the following sample code and got the below history. What could this be? I never had this problem before.
spark.sql("CREATE TABLE IF NOT EXISTS enterprisedata.hr.employees(name STRING, salary INT)")
spark.sql("DROP TABLE IF EXISTS enterprisedata.hr.employees_no_salary")
spark.sql("INSERT INTO enterprisedata.hr.employees VALUES ('Todd',10000)")
## READ TABLE IN UC
emp = spark.read.table("enterprisedata.hr.employees")
## RENAME A COLLUMN
emp = emp.withColumnRenamed("name", "name_employee" )
## DROP COLUMNS
emp = emp.drop("salary")
## WRITE
emp.write.mode("overwrite").saveAsTable("enterprisedata.hr.employees_no_salary")

