Issues with experiment errors over the last two weeks

[SamGreene]

Hi,

I use Azure Databricks in the North Central US region and have had some issues over the last two weeks.  Three weeks ago, I was able to run a forecast experiment.  

Last week I got this error on 7/24:

• [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `ds` cannot be resolved. Did you mean one of the following? [`months_between(ds, lag_d3527435, true)`]. SQLSTATE: 42703

Now I get this error in the GUI when trying to proceed past the first screen and train:

• Must provide experiment_dir where group has Manage permission when running AutoML in group clusters (POST: ajax-api/2.0/automl/experiments)

I'm using the same personal ML cluster configuration in single-user access mode.  I'm not sure what group clusters mean. 

Do you have any advice? 

Thanks!

[Kaniz_Fatma]

Hi @SamGreene, 

• This error indicates that the SQL engine cannot find the column ds. This could be due to changes in your data schema or the way the query is constructed. Ensure that the column ds exists in your dataset and is correctly referenced in your SQL query. 
• The error about needing to provide an experiment_dir suggests that your current configuration does not have the necessary permissions set up for group clusters. In Azure Databricks, group clusters are shared clusters that allow multiple users to run jobs. If you are using a personal ML cluster in single-user mode, it may not have the same permissions as a group cluster. You may need to either switch to a group cluster or adjust the permissions for your current setup.

[SamGreene]

  The ds column was definitely in the data, no doubt about it. The same cluster was used for the experiments before, so it worked a week before and then stopped. It is working again, so I guess it was just an intermittent issue.