DarrellUlm
New Contributor II

Could also use withColumn() to do it without Spark-SQL, although the performance will likely be different. The question being, would creating a new column take more time than using Spark-SQL.

Something like:

val dfNew = df.withColumn("newColName", df.originalColName.cast(IntegerType))
    .drop("originalColName").withColumnRenamed("newColName", "originalColName")

Create the new column, casting from the original column, drop the original, then rename the new column back to the original name. A bit roundabout, but looks like it could work.