Using map_filter / filter to return only map elements for a given predicate

rendorHaevyn — Wed, 11 Oct 2023 07:14:37 GMT

I want to apply a filter to a map structure (on a column called "ActivityMap") for elements only where a given predicate holds.

Representative data is below.

Applying both "map_filter" and "array_contains" will return rows where a predicate holds, however the latter returns the entire row cell contents.
eg: all elements from rows with the key "08_S" are returned:

df.select(map_filter(col("ActivityMap"), lambda k,v: array_contains(k, lit("08_S"))))

What I want returned is only the key:value elements of the map where the keys are "08_S".
eg: I want to see {"List(08_S)":[118]} or better yet, {"08_S":118}.

Perhaps the filter array function could work although I've not yet been successful with it, and clear usage examples are scant.

Any ideas on how to do this?

== single cell data map ==
{"List(08_S, 16_M, 14_O, 06_M, 12_S, 13_S, 05_P, 11_M, 10_P, 23_M, 11_S, 06_S, 01_M, 11_P, 01_S, 06_P, 17_M, 08_M, 09_S, 03_S, 05_S, 03_P, 03_M, 07_M)":[118,47,43,53,22,163,46,22,97,72,43,37,87,42,21,970,509,42,134,115,26,143,20,20]}

topic Using map_filter / filter to return only map elements for a given predicate in Get Started Discussions

Using map_filter / filter to return only map elements for a given predicate