Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

RiyazAliM — Tue, 21 Jun 2022 08:42:00 GMT

HI there,

I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy.

In ML-08 - Hyperopt notebook, I see the following error in cmd 13.

  best_hyperparam = fmin(fn=objective_function, 
                         space=search_space,
                         algo=tpe.suggest, 
                         max_evals=num_evals,
                         trials=trials,
#this is where I see the error, I commented the randomstate to bypass the error.
                         rstate=np.random.RandomState(42) 
                        )

The below is the error i've encountered:

AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Detailed Error:

AttributeError                            Traceback (most recent call last)
<command-1903711140984667> in <module>
      6   num_evals = 4
      7   trials = Trials()
----> 8   best_hyperparam = fmin(fn=objective_function, 
      9                          space=search_space,
     10                          algo=tpe.suggest,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    563 
    564     if allow_trials_fmin and hasattr(trials, "fmin"):
--> 565         return trials.fmin(
    566             fn,
    567             space,
 
/databricks/.python_edge_libs/hyperopt/base.py in fmin(self, fn, space, algo, max_evals, timeout, loss_threshold, max_queue_len, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar, early_stop_fn, trials_save_file)
    669         from .fmin import fmin
    670 
--> 671         return fmin(
    672             fn,
    673             space,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    609 
    610     # next line is where the fmin is actually executed
--> 611     rval.exhaust()
    612 
    613     if return_argmin:
 
/databricks/.python_edge_libs/hyperopt/fmin.py in exhaust(self)
    387     def exhaust(self):
    388         n_done = len(self.trials)
--> 389         self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
    390         self.trials.refresh()
    391         return self
 
/databricks/.python_edge_libs/hyperopt/fmin.py in run(self, N, block_until_done)
    297                     # processes orchestration
    298                     new_trials = algo(
--> 299                         new_ids, self.domain, trials, self.rstate.integers(2 ** 31 - 1)
    300                     )
    301                     assert len(new_ids) >= len(new_trials)
 
AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Another error is in ML08L-Hyperopt Lab, where I'm unable to pass I'm unable to pass "max_features" from best_hyperparam to the regressor. It fails while I'm fitting the model to the X_train & y_train.

Below is the snip:

The error statement reads:

ValueError: max_features must be in (0, n_features]

Re: Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

RiyazAliM — Wed, 22 Jun 2022 12:51:15 GMT

Tagging @Kaniz Fatma as there was no response what so ever!

By any chance, do you know how to resolve these errors in the notebook?

Thanks!

topic Re: Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy. in Data Engineering

Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

Re: Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.