AutoML "need to sample" not working as expected

sangramraje — Fri, 22 Nov 2024 18:33:12 GMT

tl; dr:

When the AutoML run realizes it needs to do sampling because the driver / worker node memory is not enough to load / process the entire dataset, it fails. A sample weight column is NOT provided by me, but I believe somewhere in the process the automl system believes it was supplied, tries to find it and encounters an error:

[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `_automl_sample_weight_0000` cannot be resolved. Did you mean one of the following? [`_automl_split_col_0000`, `t*****d`, `o******l`, `s*******o`, `f*******o`]. SQLSTATE: 42703

More details:

I am running an experiment with AutoML on 15.4LTS ML runtime cluster. I set up the experiment with a driver node i3.2xlarge and worker node r5d.2xlarge. I see a log in the AutoML run:

2024/11/22 17:41:53 INFO databricks.automl.internal.size_estimator: mem_req_data_load_mb = 52148.01129505962 mem_req_training_mb_dense = 30748.212867736816 mem_req_training_mb_sparse = 28898.903835296627 mem_req_training_mb = 30748.212867736816 2024/11/22 17:41:53 INFO databricks.automl.internal.size_estimator: fraction (0.34224123905739046) = min of (available_memory_mb_per_trial (41139.0) / worker_max_memory_req_mb (52148.01129505962)), (self._memory_mb_on_driver (17847.199999999997) / mem_req_data_load_mb (52148.01129505962))

At this point, it goes into the code where it seems it is trying to sample and then fails. I am attaching the stacktrace as a pdf.

See below snippet from the log showing that the sample_weight_col is NOT provided:

2024/11/22 17:38:56 INFO databricks.automl.internal.supervised_learner: AutoML called with params: target_col=w***s, data_dir=None exclude_cols=None exclude_columns=['p***d', 'E***d', 'T***D', 'M***e', 'T***y'] exclude_frameworks=['lightgbm', 'sklearn'] imputers=None metric=roc_auc max_trials=10000000000.0 timeout_minutes=120 experiment_id=4***6 time_col=None experiment_dir=/Users/s***m/databricks_automl pos_label=1 split_col=None sample_weight_col=None

The 2nd image above (Classifier._sample) clearly shows at that point the `sample_weight_col` is not None (goes inside the if). The 1st image is what seems suspicious to me, hence adding that.

Any help is most appreciated! I believe this piece of the source code is "internal" and closed hence posting here is the best I could do.

Re: AutoML "need to sample" not working as expected

Louis_Frolio — Thu, 30 Oct 2025 09:59:22 GMT

Hey @sangramraje , sorry for the late response. I wanted to check in to see if this is still an issue with the latest release? Please let me know. Cheers, Louis.

topic AutoML "need to sample" not working as expected in Machine Learning

AutoML "need to sample" not working as expected

Re: AutoML "need to sample" not working as expected