You have already mentioned you did turn off autoscaling, please try the num_workers too

Step 1: Disable Dynamic Resource Allocation: Use spark.dynamicAllocation.enabled = false

Step 2: Configure num_workers to Match Fixed Resources

After disabling dynamic allocation, you must ensure that the number of workers requested by SparkXGBRanker does not exceed the actual number of executor cores available, and ideally, should match them.

a. Total Cores=(Number of Worker Nodes)×(Cores per Worker)

b. Then, set the num_workers parameter in your SparkXGBRanker constructor to a value less than or equal to the total available cores.

# Get the number of available cores (or set a fixed number of partitions)
num_partitions = data.rdd.getNumPartitions()

# Instantiate the ranker, setting num_workers equal to the number of partitions
# or the total cores on the cluster, ensuring it is a fixed number.
ranker = SparkXGBRanker(
    num_workers=num_partitions, # Or a number matching your cluster's total cores
    ... # other parameters
)

pipeline_model = pipeline.fit(data)