def df_process_function(df):
    schema = df.schema
    def partition_process(iterator):
        for pdf in iterator:
            processed_records = partition_function(pdf)
            yield pd.DataFrame(processed_records)
    return df.mapInPandas(process_partition, schema=schema)

But the column that is shown in the error is different from that is modified.

The schema of the spark dataframe is not modified dynamically.