Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2017 12:26 PM
Could also use withColumn() to do it without Spark-SQL, although the performance will likely be different. The question being, would creating a new column take more time than using Spark-SQL.
Something like:
val dfNew = df.withColumn("newColName", df.originalColName.cast(IntegerType))
.drop("originalColName").withColumnRenamed("newColName", "originalColName")
Create the new column, casting from the original column, drop the original, then rename the new column back to the original name. A bit roundabout, but looks like it could work.