Hi,

1. we changed our large string column to an array column

2. we use a bigger instance for the driver

So we can handle 500 files in our notebook transformation.

This runs the most time, but sporadically the same error occurs.

I could't change our UDF to pandas_udf. I do not really understand how to change it to an panas_udf.

Maybe you could help me, i would e.g. change this udf to a pandas_udf:

(Here we decode a binary content to a string content)

  def decode_content(self, input_df: DataFrame) -> DataFrame:

    print("Decode binary content to string")

    # udf

    def decode_content(s):

      gzip = base64.b64decode(s)

      gzip1 = gzip[4:]

      return decompress(gzip1).decode()

    # create udf

    decode_content_udf = udf(lambda z: decode_content(z), StringType())

    # use udf

    result = input_df.withColumn("decoded", decode_content_udf(input_df.content))

    result = result.select(col("decoded"))

    return result