I solved this by defining the applyInPandas transformation inside of the function that it's used in. 

 


def apply_coordinate_transform(data):
    def coordinate_transform(group_keys, pdf) -> pd.DataFrame:
    
        trafo = get_coordinate_transformation(group_keys[0])
        ... do some pandas code here
        return pdf
    ...
    schema = data.schema
    data = data.groupBy('serialnumber',...)\
               .applyInPandas(coordinate_transform, schema=schema)

    return data

 

 

View solution in original post