OK.

Basically you should never loop over a dataframe because that renders the distributed capacity of Spark useless.

what you should do is:

1. read the delta table into a dataframe with spark.read.table(table)
2. then do your transformations. Updating a column is done with the withColumn() statement. There are tons of other functions of course.
3. finally write the data. This can be either in append (as you did), merge (upsert) or overwrite (replace all).

There are some interesting tutorials on the databricks website which give an introduction to spark/databricks.