I am facing the below error while fitting my model. I am trying to run a model with cross validation with a pipeline inside of it. Below is the code snippet for data transformation:

qd = QuantileDiscretizer(relativeError=0.01, handleInvalid="error", numBuckets=4, 
    inputCols=["time"], outputCols=["time_qd"])
 
    #Normalize Vector
    scaler = StandardScaler()\
             .setInputCol ("vectorized_features")\
             .setOutputCol ("features")
 
    #Encoder for VesselTypeGroupName
    encoder = StringIndexer(handleInvalid='skip')\
        .setInputCols (["type"])\
        .setOutputCols (["type_enc"])
 
    #OneHot encoding categorical variables
    encoder1 = OneHotEncoder()\
        .setInputCols (["type_enc", "ID1", "ID12", "time_qd"])\
        .setOutputCols (["type_enc1", "ID1_enc", "ID12_enc", "time_qd_enc"])
 
    #Assembling Variables
    assembler = VectorAssembler(handleInvalid="keep")\
             .setInputCols (["type_enc1", "ID1_enc", "ID12_enc", "time_qd_enc"]) \
             .setOutputCol ("vectorized_features")

The total number of features after one hot encoding will not exceed 200. The model code is below:

lr = LogisticRegression(featuresCol = 'features', labelCol = 'label', 
    weightCol='classWeightCol')
    pipeline_stages = Pipeline(stages=[qd , encoder, encoder1 , assembler , scaler, lr])
    #Create Logistic Regression parameter grids for parameter tuning
    paramGrid_lr = (ParamGridBuilder()
                 .addGrid(lr.regParam, [0.01, 0.5, 2.0])# regularization parameter
                 .addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])# Elastic Net Parameter (Ridge = 0)
                 .addGrid(lr.maxIter, [1, 10, 20])# Number of iterations
                 .build())
    cv_lr = CrossValidator(estimator=pipeline_stages, estimatorParamMaps=paramGrid_lr, 
                        evaluator=BinaryClassificationEvaluator(), numFolds=5, seed=42)
    cv_lr_model = cv_lr.fit(train_df)

.fit method throws the below error:

I have tried increasing the driver memory (28GB ram with 8 cores) but still facing the same error. Please help what is the cause of this issue.