The problem was indeed with the way ClassLoader was being set in the ForkJoinPool (common Pool used) thread. Spark in SparkClassUtils uses Thread.currentThread().getContextClassLoader which might behave differently in another thread.

To solve it I created my own ForkJoinWorkerThreadFactory implementation and set the ContextClassLoader using that of the Main thread. Then created my own custom ForkJoinPool to execute the parallelStream.

public class MyForkJoinWorkerThreadFactory implements ForkJoinWorkerThreadFactory {

  @Override
  public ForkJoinWorkerThread newThread(ForkJoinPool pool) {
    final ForkJoinWorkerThread worker = ForkJoinPool.defaultForkJoinWorkerThreadFactory.newThread(pool);
    worker.setName("my-fjpworker-" + worker.getPoolIndex());
    worker.setContextClassLoader(Thread.currentThread().getContextClassLoader());  // Set the ClassLoader
    return worker;
  }
}

Similar issue: JAXB class not found with ForkJoinPool Java

Code reference: https://stackoverflow.com/a/57551188/23113106