Select dataframe columns from a sequence of string

Jean-FrancoisRa — Wed, 25 Nov 2015 21:10:14 GMT

Is there a simple way to select columns from a dataframe with a sequence of string?

Something like

val colNames = Seq("c1", "c2")
df.select(colNames)

Re: Select dataframe columns from a sequence of string

JongKim — Thu, 03 Dec 2015 02:54:33 GMT

I also had the same problem, and here's how to make it work using column type and varargs:

// make example dataframe import org.apache.spark.sql.DataFrame val df: DataFrame = sc.parallelize(Seq((1, 2, 3), (4, 5, 6), (7, 8, 9))).toDF("a", "b", "c")

// desired list of column names in string (making it possible programmatically) val column_names_str = Seq[String]("a", "b")

// construct list of column names in column type import org.apache.spark.sql.functions.col val column_names_col = column_names_str.map(name => col(name)) //val column_names_col = column_names_str.map(name => col(name).as(s"renamed_$name")) // rename if needed

// select specific columns from dataframe using varargs syntax * val df_new = df.select(column_names_col: *) df_new.show()

This should yield as expected:

+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  4|  5|
|  7|  8|
+---+---+

Re: Select dataframe columns from a sequence of string

vEdwardpc — Sat, 01 Oct 2016 20:21:41 GMT

Thanks. I needed to modify the final lines.

val df_new = df.select(column_names_col:_*)
df_new.show()

Edward

topic Re: Select dataframe columns from a sequence of string in Data Engineering

Select dataframe columns from a sequence of string

Re: Select dataframe columns from a sequence of string

Re: Select dataframe columns from a sequence of string