lumen
New Contributor II

Yeah in my case I'm starting to think this could be an installation thing, or a unset property. So I created the following sample code and got the below history. What could this be? I never had this problem before.

spark.sql("CREATE TABLE IF NOT EXISTS enterprisedata.hr.employees(name STRING, salary INT)")
spark.sql("DROP TABLE IF EXISTS enterprisedata.hr.employees_no_salary")
spark.sql("INSERT INTO enterprisedata.hr.employees VALUES ('Todd',10000)")

## READ TABLE IN UC
emp = spark.read.table("enterprisedata.hr.employees")

## RENAME A COLLUMN
emp = emp.withColumnRenamed("name", "name_employee" )

## DROP COLUMNS
emp = emp.drop("salary")

## WRITE
emp.write.mode("overwrite").saveAsTable("enterprisedata.hr.employees_no_salary")

Image 23-05-2025 at 19.08.pngImage 23-05-2025 at 19.09.png