AutoML: 403 Error error_code:"PERMISSION_DENIED"

[spearitchmeta]

Hello Community,

I am using AutoML on my AutoML enabled cluster and have a service principle. Here is the code

features_for_split = df.drop(columns=['cae_type', 'id']).select_dtypes(include=[np.number])
target_for_split = df['cae_type']

# Use same random state as client's code for identical split
X_train, X_test, y_train, y_test = train_test_split(
    features_for_split, target_for_split, 
    test_size=0.2, 
    random_state=42,
    # Ensure balanced splits for imbalanced data
    #stratify=target_for_split
)

# Create training DataFrame for AutoML
train_df = X_train.copy()
train_df['cae_type'] = y_train

# Configure AutoML experiment
experiment_auto_ml_manual_split = f"CAE_MANUAL_split_{datetime.now().strftime('%Y%m%d_%H%M%S')}"

# Run AutoML with client's data split
automl_result = automl.classify(
    dataset=train_df,
    target_col="cae_type",
    primary_metric="f1",
    timeout_minutes=180,
    experiment_dir=experiment_dir,
    experiment_name=experiment_name
)

I am getting the following error where experiment_dir /Workspace/Users/myeamail@domain.com/
AutomlServiceError: Failed to create Automl experiment. Status Code: 403 Error: b'{"error_code":"PERMISSION_DENIED","message":"MY_SERVICE_PRINCIPLE does not have View permissions on 0. Please contact the owner or an administrator for access.","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"a53d66d5-ff67-43f4-97e5-bd95b5de46d0","serving_data":""}]}'

According to chatgpt and to the article https://kb.databricks.com/machine-learning/permission_denied-error-while-running-automl-experiment-w...  

• Group-assigned clusters always run as the group principal, not as your individual user.

  • Anything the cluster writes or reads must therefore live in a location where the group has at least “CAN MANAGE” (or “CAN EDIT”) permissions.

• You created /Workspace/Users/myemail@domain.com/…, which your user can reach—but the cluster’s group cannot.

• AutoML tries to create the MLflow experiment inside experiment_dir. Because the group principal can’t see that folder, MLflow’s underlying “create experiment” call is rejected with

The article is clear.. create a Group folder under Workspace and assign the correct permission to it.

group_name = "XXXX"

experiment_dir = f"/Workspace/Groups/{group_name}/automl_experiments"

experiment_name = f"manual_split_{datetime.now():%Y%m%d_%H%M%S}"

dbutils.fs.mkdirs(experiment_dir)

Yet I am not being able top see the created folder "Groups/{group_name}" under Workspace. Is this due to my lack of Admin Rights? Because when I check whether the folders exist, the response is positive

[szymon_dybczak · Accepted solution]

Hi @spearitchmeta ,

When you're using following code, you're interacting wiht dbfs and there are several limitations that apply to dbutils when it comes to interacting with Workspace files.. 

dbutils.fs.mkdirs(experiment_dir)

 It's confusing but you can read about the difference here:

Databricks Utilities (dbutils) reference | Databricks Documentation

So, your folder should exists but not in the place where you wanted it to be. I've recreated your example, check screenshot below:

As you can see, the folder was created but under Workspace directory that resides in dbfs (which is different from workspace files you see in UI).

You wanted to create a directory in a Workspace. You can do it through UI or using %sh:

%sh
mkdir -p /Workspace/Groups/XXXX/automl_experiments

After executing above command, new directory appeared in my workspace:

Of course, when you create this new folder you need to assing proper permission to it for you group - namely CAN MANAGE permission.

Once you perform above steps your code should work.

Edit: Make sure that whatever group you are using has the "Workspace Access" entitlement enabled.