Serverless job error - spark.rpc.message.maxSize

adurand-accure
Databricks Partner

Hello, 

I am facing this error when moving a Workflow to serverless mode

ERROR : SparkException: Job aborted due to stage failure: Serialized task 482:0 was 269355219 bytes, which exceeds max allowed: spark.rpc.message.maxSize (268435456 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

on JOB cluster we could set the  spark.rpc.message.maxSize manually to a value greater than 268 m, which looks not possible on Serverless


Any help is appreciated, thx