Append to a record when updating Delta Table

Serhii
Contributor

I am updating the delta table in Databricks as follows

    segments_data.alias('segments_old').merge(
        segments_data_new.alias("updates"),
        "segments_old.source_url = updates.source_url",
    ).whenMatchedUpdate(
        set={"segments": "segments_old.segments"  + "updates.segments"  } # <- the line is pseudocode, don't know correct API
    ).whenNotMatchedInsertAll().execute()

where "segments" is a list of StructType

    StructField(
      "segments",
      StructType(
        [ ....

How can I append to the existing list - namely append to "segments" column in segments_old from updates within "whenMatchedUpdate"?

Instead of replacing - say "segments": "updates.segments" I would like to append "segments": "segments_old.segments" + "updates.segments"

Thanks for suggestions in advance!