How do we concatenate two columns in an Apache Spark DataFrame? Is there any function in Spark SQL which we can use?

[saipujari_spark · Accepted solution]

Yes, we can concat() and concat_ws() inbuilt functions.

concat - usage

> SELECT concat('Spark', 'SQL');
 SparkSQL

concat_ws - usage - concatenates with a separator

SELECT concat_ws(' -', 'Spark', 'SQL');
  Spark-SQL

Reference: https://spark.apache.org/docs/latest/api/sql/index.html