topic Re: issue with rounding selected column in "for in" loop in Data Engineering

issue with rounding selected column in "for in" loop

Braxx — Mon, 14 Feb 2022 10:40:38 GMT

This must be trivial, but I must have missed something.

I have a dataframe (test1) and want to round all the columns listed in list of columns (col_list)

here is the code I am running:

col_list = ['measure1', 'measure2', 'measure3']
 
for i in col_list:
  rounding = test1\
    .withColumn(i, round(col(i),0))
 
display(rounding)

and as a result only the last column has its values rounded.

What am I missing to have all the measures rounded?

data for testing:

car model measure1 measure2 measure3

Nissan aa 1.11 0.3 34

Toyota bb 1.12 0.4 111

BMW cc 1.13 0.5 1.9

Re: issue with rounding selected column in "for in" loop

AmanSehgal — Mon, 14 Feb 2022 13:27:43 GMT

Because it's a for loop, and in the last loop column measure3 is selected. The variable rounding is assigned a new dataframe with changes that occur on column measure3 only.

Try the following code:

rounding = test1
for i in col_list:
  rounding = rounding\
    .withColumn(i, round(col(i),0))

Re: issue with rounding selected column in "for in" loop

Braxx — Mon, 14 Feb 2022 14:50:07 GMT

You're absolutely right. thanks