AutoML: 403 Error error_code:"PERMISSION_DENIED"

spearitchmeta — Fri, 08 Aug 2025 12:06:38 GMT

Hello Community,

I am using AutoML on my AutoML enabled cluster and have a service principle. Here is the code

features_for_split = df.drop(columns=['cae_type', 'id']).select_dtypes(include=[np.number]) target_for_split = df['cae_type'] # Use same random state as client's code for identical split X_train, X_test, y_train, y_test = train_test_split( features_for_split, target_for_split, test_size=0.2, random_state=42, # Ensure balanced splits for imbalanced data #stratify=target_for_split ) # Create training DataFrame for AutoML train_df = X_train.copy() train_df['cae_type'] = y_train # Configure AutoML experiment experiment_auto_ml_manual_split = f"CAE_MANUAL_split_{datetime.now().strftime('%Y%m%d_%H%M%S')}" # Run AutoML with client's data split automl_result = automl.classify( dataset=train_df, target_col="cae_type", primary_metric="f1", timeout_minutes=180, experiment_dir=experiment_dir, experiment_name=experiment_name )

I am getting the following error where experiment_dir /Workspace/Users/myeamail@domain.com/
AutomlServiceError: Failed to create Automl experiment. Status Code: 403 Error: b'{"error_code":"PERMISSION_DENIED","message":"MY_SERVICE_PRINCIPLE does not have View permissions on 0. Please contact the owner or an administrator for access.","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"a53d66d5-ff67-43f4-97e5-bd95b5de46d0","serving_data":""}]}'

According to chatgpt and to the article https://kb.databricks.com/machine-learning/permission_denied-error-while-running-automl-experiment-with-group-assigned-cluster

Group-assigned clusters always run as the group principal, not as your individual user.
- Anything the cluster writes or reads must therefore live in a location where the group has at least “CAN MANAGE” (or “CAN EDIT”) permissions.
You created /Workspace/Users/myemail@domain.com/…, which your user can reach—but the cluster’s group cannot.
AutoML tries to create the MLflow experiment inside experiment_dir. Because the group principal can’t see that folder, MLflow’s underlying “create experiment” call is rejected with

The article is clear.. create a Group folder under Workspace and assign the correct permission to it.

group_name = "XXXX"

experiment_dir = f"/Workspace/Groups/{group_name}/automl_experiments"

experiment_name = f"manual_split_{datetime.now():%Y%m%d_%H%M%S}"

dbutils.fs.mkdirs(experiment_dir)

Yet I am not being able top see the created folder "Groups/{group_name}" under Workspace. Is this due to my lack of Admin Rights? Because when I check whether the folders exist, the response is positive

Re: AutoML: 403 Error error_code:"PERMISSION_DENIED"

szymon_dybczak — Fri, 08 Aug 2025 13:37:49 GMT

Hi @spearitchmeta ,

When you're using following code, you're interacting wiht dbfs and there are several limitations that apply to dbutils when it comes to interacting with Workspace files..

dbutils.fs.mkdirs(experiment_dir)

It's confusing but you can read about the difference here:

Databricks Utilities (dbutils) reference | Databricks Documentation

So, your folder should exists but not in the place where you wanted it to be. I've recreated your example, check screenshot below:

As you can see, the folder was created but under Workspace directory that resides in dbfs (which is different from workspace files you see in UI).

You wanted to create a directory in a Workspace. You can do it through UI or using %sh:

%sh mkdir -p /Workspace/Groups/XXXX/automl_experiments

After executing above command, new directory appeared in my workspace:

Of course, when you create this new folder you need to assing proper permission to it for you group - namely CAN MANAGE permission.

Once you perform above steps your code should work.

Edit: Make sure that whatever group you are using has the "Workspace Access" entitlement enabled.

topic AutoML: 403 Error error_code:"PERMISSION_DENIED" in Machine Learning

AutoML: 403 Error error_code:"PERMISSION_DENIED"

Re: AutoML: 403 Error error_code:"PERMISSION_DENIED"