Hello!
I'm attempting to run a hyperparameter search using hyperopt and SparkTrials(), and log the resulting runs to an existing experiment (experiment A). I can see on this page that databricks suggests wrapping the `fmin()` call within a `mlflow.start_run()` statement, so I wrote this code in a notebook, which by default is associated with its own experiment (experiment B):
with mlflow.start_run(experiment_id=experiment_A):
trials = SparkTrials()
fmin(objective_fn,
trials=trials,
**other_params)
However, the result I get is a single parent run logged in experiment_A with all its child runs logged in experiment_B. This is obviously not desirable. Is there any way I can get both the parent and child runs from the trials logged in experiment A (i.e., not the experiment associated with the notebook being run)?
