To ensure that MLflow child runs appear as children of their parent run in the MLflow GUI when using a custom experiment location, follow these steps:

1. Set Up the Experiment Location:

   EXPERIMENT_NAME = '/Users/dxxxx@realchemistry.com/MLflow_experiments/dxxxx_minimal_MLflow'
   
   # Get the experiment ID if it exists, or create a new one
   experiment_id = mlflow.get_experiment_by_name(EXPERIMENT_NAME)
   
   if experiment_id is None:
       # If the experiment does not exist, create it
       experiment_id = mlflow.create_experiment(EXPERIMENT_NAME)
   else:
       # If the experiment exists, get its ID
       experiment_id = experiment_id.experiment_id

2. Start the Parent Run:

   with mlflow.start_run(experiment_id=experiment_id, run_name='xgboost_models_2') as parent_run:
       run_id_value = parent_run.info.run_id
       search_space['parent_run_id'] = run_id_value
       best_params = fmin(
           fn=train_model, 
           space=search_space, 
           algo=tpe.suggest, 
           max_evals=8,
           trials=spark_trials,
       )

3. Define the Training Function with Nested Runs:

   def train_model(params):
       mlflow.xgboost.autolog()
       with mlflow.start_run(nested=True):
           train = xgb.DMatrix(data=X_train, label=y_train)
           validation = xgb.DMatrix(data=X_val, label=y_val)
           # Additional training code here

4. Ensure Correct Parent-Child Relationship:

   • Verify that the parent_run_id is correctly set in the search_space.
   • Ensure that the nested=True parameter is used in the mlflow.start_run call within the train_model function.