We are exploding a map type column into multiple columns based on the keys of the map column. Part of this process is to extract the keys of a map type column called json_map as illustrated in the snippet below. The code executes as expected when running it in a notebook, but returns an empty array when running it in a Delta Live Tables pipeline. Below is the snippet of code:

    keys = (
        df
        .select(map_keys("json_map"))
        .distinct()
        .collect()
        )

Does anyone know why this code will run as expected in notebook, but returns empty array in the Delta Live Tables pipeline? Or is there another method to extract the different fields of a map or json column to separate columns?