Dooley
Databricks Employee
Databricks Employee

Try the below:

def CNN_HOF(train_df_pd, valid_df_pd, test_df_pd, params): #Hyperopt objective function
    train_generator = train_data_gen.flow_from_dataframe(dataframe=train_df_pd,
                                                         directory=images_dir,
                                                         x_col='filename',
                                                         y_col=target,
                                                         target_size=(150, 150),
                                                         class_mode='categorical',
                                                         batch_size=train_batch)
     valid_generator = valid_data_gen.flow_from_dataframe(dataframe=valid_df_pd,
                                                         directory=images_dir,
                                                         x_col='filename',
                                                         y_col=target,
                                                         target_size=(150, 150),
                                                         class_mode='categorical',
                                                         batch_size=valid_batch,
                                                         shuffle=False,
                                                         seed=42)
    test_generator = test_data_gen.flow_from_dataframe(dataframe=test_df_pd,
                                                           directory=images_dir,
                                                           x_col='filename',
                                                           y_col=target,
                                                           target_size=(150, 150),
                                                           class_mode='categorical',
                                                           batch_size=test_batch,
                                                           shuffle=False,
                                                           seed=42)
    mlflow.tensorflow.autolog()
    model = model_builder(params,dense_size)
    model.compile(loss="categorical_crossentropy",
                optimizer=Adam(),
                metrics=["accuracy"])
 
    history = model.fit(train_generator,
                        steps_per_epoch=train_step,
                        epochs=tuner_epochs,
                        validation_data=valid_generator,
                        validation_steps=valid_step,
                        verbose=2)
  # Evaluate the model
    score = model.evaluate(test_generator, steps=1, verbose=0)
    obj_metric = score[0]
    return float(obj_metric)

I'm assuming your dense_size, valid_batch, train_batch, test_batch, image_dir, and target are global variables. Note that train_df_pd = train_df.toPandas(), valid_df_pd = valid_df.toPandas(), and test_df_pd = test_df.toPandas() are outside of the objective function and the pandas dataframes are being brought in as arguments. Then you put the generator in the objective function in here. If this works, then the generator was the issue and we can do something to speed up this process.

I also want to note that I am returning the score as a float. I'm wondering if that score[0] is a dictionary and not a float but I casted it but you can print that type to validate.

View solution in original post