topic column wise sum in PySpark dataframe in Data Engineering

column wise sum in PySpark dataframe

siddhu308 — Mon, 22 Apr 2019 08:36:13 GMT

i have a dataframe of 18000000rows and 1322 column with '0' and '1' value.

want to find how many '1's are in every column ???

below is DataSet

se_00001 se_00007 se_00036 se_00100 se_0010p se_00250

siddhu308 — Mon, 22 Apr 2019 08:37:04 GMT

seag_00001 seag_00007 seag_00036 seag_00100 seag_0010p seag_00250

1 0 1 0 0 0

mathan_pillai — Tue, 23 Apr 2019 14:41:14 GMT

Hi Siddhu,

You can use


df.select(sum("col1"), sum("col2"), sum("col3"))
where col1, col2, col3 are the column names for which you would like to find the sum

please let us know if it answers your question

Thanks