Serverless job error - spark.rpc.message.maxSize
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2024 03:42 AM
Hello,
I am facing this error when moving a Workflow to serverless mode
ERROR : SparkException: Job aborted due to stage failure: Serialized task 482:0 was 269355219 bytes, which exceeds max allowed: spark.rpc.message.maxSize (268435456 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.
on JOB cluster we could set the spark.rpc.message.maxSize manually to a value greater than 268 m, which looks not possible on Serverless
Any help is appreciated, thx