Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

[RiyazAli]

HI there,

I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy.

In ML-08 - Hyperopt notebook, I see the following error in cmd 13.

  best_hyperparam = fmin(fn=objective_function, 
                         space=search_space,
                         algo=tpe.suggest, 
                         max_evals=num_evals,
                         trials=trials,
#this is where I see the error, I commented the randomstate to bypass the error.
                         rstate=np.random.RandomState(42) 
                        )

The below is the error i've encountered:

AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Detailed Error:

AttributeError                            Traceback (most recent call last)
<command-1903711140984667> in <module>
      6   num_evals = 4
      7   trials = Trials()
----> 8   best_hyperparam = fmin(fn=objective_function, 
      9                          space=search_space,
     10                          algo=tpe.suggest,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    563 
    564     if allow_trials_fmin and hasattr(trials, "fmin"):
--> 565         return trials.fmin(
    566             fn,
    567             space,
 
/databricks/.python_edge_libs/hyperopt/base.py in fmin(self, fn, space, algo, max_evals, timeout, loss_threshold, max_queue_len, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar, early_stop_fn, trials_save_file)
    669         from .fmin import fmin
    670 
--> 671         return fmin(
    672             fn,
    673             space,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    609 
    610     # next line is where the fmin is actually executed
--> 611     rval.exhaust()
    612 
    613     if return_argmin:
 
/databricks/.python_edge_libs/hyperopt/fmin.py in exhaust(self)
    387     def exhaust(self):
    388         n_done = len(self.trials)
--> 389         self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
    390         self.trials.refresh()
    391         return self
 
/databricks/.python_edge_libs/hyperopt/fmin.py in run(self, N, block_until_done)
    297                     # processes orchestration
    298                     new_trials = algo(
--> 299                         new_ids, self.domain, trials, self.rstate.integers(2 ** 31 - 1)
    300                     )
    301                     assert len(new_ids) >= len(new_trials)
 
AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Another error is in ML08L-Hyperopt Lab, where I'm unable to pass I'm unable to pass "max_features" from best_hyperparam to the regressor. It fails while I'm fitting the model to the X_train & y_train.

Below is the snip:

The error statement reads:

ValueError: max_features must be in (0, n_features]

[RiyazAli]

Tagging @Kaniz Fatma​ as there was no response what so ever!

By any chance, do you know how to resolve these errors in the notebook?

Thanks!