cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

RiyazAli
Valued Contributor

HI there,

I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy.

In ML-08 - Hyperopt notebook, I see the following error in cmd 13.

hyperopt_implementation

  best_hyperparam = fmin(fn=objective_function, 
                         space=search_space,
                         algo=tpe.suggest, 
                         max_evals=num_evals,
                         trials=trials,
#this is where I see the error, I commented the randomstate to bypass the error.
                         rstate=np.random.RandomState(42) 
                        )

The below is the error i've encountered:

AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Detailed Error:

AttributeError                            Traceback (most recent call last)
<command-1903711140984667> in <module>
      6   num_evals = 4
      7   trials = Trials()
----> 8   best_hyperparam = fmin(fn=objective_function, 
      9                          space=search_space,
     10                          algo=tpe.suggest,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    563 
    564     if allow_trials_fmin and hasattr(trials, "fmin"):
--> 565         return trials.fmin(
    566             fn,
    567             space,
 
/databricks/.python_edge_libs/hyperopt/base.py in fmin(self, fn, space, algo, max_evals, timeout, loss_threshold, max_queue_len, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar, early_stop_fn, trials_save_file)
    669         from .fmin import fmin
    670 
--> 671         return fmin(
    672             fn,
    673             space,
 
/databricks/.python_edge_libs/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, timeout, loss_threshold, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar, early_stop_fn, trials_save_file)
    609 
    610     # next line is where the fmin is actually executed
--> 611     rval.exhaust()
    612 
    613     if return_argmin:
 
/databricks/.python_edge_libs/hyperopt/fmin.py in exhaust(self)
    387     def exhaust(self):
    388         n_done = len(self.trials)
--> 389         self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
    390         self.trials.refresh()
    391         return self
 
/databricks/.python_edge_libs/hyperopt/fmin.py in run(self, N, block_until_done)
    297                     # processes orchestration
    298                     new_trials = algo(
--> 299                         new_ids, self.domain, trials, self.rstate.integers(2 ** 31 - 1)
    300                     )
    301                     assert len(new_ids) >= len(new_trials)
 
AttributeError: 'numpy.random.mtrand.RandomState' object has no attribute 'integers'

Another error is in ML08L-Hyperopt Lab, where I'm unable to pass I'm unable to pass "max_features" from best_hyperparam to the regressor. It fails while I'm fitting the model to the X_train & y_train.

Below is the snip:

hyperopt problem with &quot;max_features&quot; The error statement reads:

ValueError: max_features must be in (0, n_features]

2 REPLIES 2

RiyazAli
Valued Contributor

Tagging @Kaniz Fatma​ as there was no response what so ever!

By any chance, do you know how to resolve these errors in the notebook?

Thanks!

Kaniz
Community Manager
Community Manager

Hi @Riyaz Ali​, Thank you for reaching out!

Let us look into this for you, and we'll circle back with an update.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!