Dooley
Databricks Employee
Databricks Employee

From looking through your full error, it looks like the error is first with FMinIter before it tries to do a Pickler dump after that error which is what is throwing that specific error. I think the true error we are chasing here is the FMinIter error.

First, try to test the search space on something simpler (maybe only trying it only pool for one layer at first). Additionally, I would use hp.quniform with min >>= 1 such as this:

search_space = {
    "pool_1": hp.quniform("pool_1", 2, 5, 1),
}

Another thought is that you need to load your train_generator & test_generator in the objective function which I'm assuming could look like this:

train_generator = datagen.flow_from_directory(directory="/image/path", 
                                              class_mode="binary", 
                                              classes=["normal", "abnormal"],
                                              batch_size=batch_size,
                                              target_size=(img_height, img_width))