topic Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST in Machine Learning

ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 08:35:05 GMT

Followed the below documentation to create a ML experiment -
https://docs.databricks.com/aws/en/mlflow/experiments

I created an experiment using the databricks console, then tried running the below code but getting error - getting error - RESOURCE_DOES_NOT_EXIST: Parent directory /Users/<username> does not exist

import mlflow

import os

import numpy as np

from sklearn.linear_model import LinearRegression

experiment_name = "/Users/<username>/my_ml_experiment"

mlflow.set_experiment(experiment_name)

with mlflow.start_run():

mlflow.log_param("alpha", 0.01)

mlflow.log_param("fit_intercept", True)

mlflow.log_metric("rmse", 0.25)

mlflow.log_metric("r2", 0.95)

artifact_dir = "/dbfs/FileStore/mlflow_artifacts"

os.makedirs(artifact_dir, exist_ok=True)

artifact_path = os.path.join(artifact_dir, "info.txt")

with open(artifact_path, "w") as f:

f.write("This is a sample artifact for MLflow logging in Databricks.")

mlflow.log_artifact(artifact_path)

X = np.array([[1], [2], [3], [4]])

y = np.array([2, 4, 6, 8])

model = LinearRegression(fit_intercept=True)

model.fit(X, y)

mlflow.sklearn.log_model(model, "linear_model")

experiment = mlflow.get_experiment_by_name(experiment_name)

print(f"Experiment ID: {experiment.experiment_id}")

print(f"Artifact Location: {experiment.artifact_location}")

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

BS_THE_ANALYST — Mon, 11 Aug 2025 08:38:42 GMT

@dbuser24 I think this is just an innocent oversight.

In your code you have the following:
experiment_name = "/Users/<username>/my_ml_experiment"

I'm assuming you were meant to fill this in and reference a directory that you have available in Databricks, or create one for this exercise. That's why the error is saying "RESOURCE_DOES_NOT_EXIST: Parent directory /Users/<username> does not exist" .. it can't find this location.

All the best,
BS

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 09:04:05 GMT

@BS_THE_ANALYST Thanks for your response, there exists a directory with my username, I have just used a placeholder <username> due to company privacy policy. I have also tried the Shared direc

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 09:04:53 GMT

Does ML experiments need some king of setting at the admin level ? As, other members in the team are also getting the same error.

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

BS_THE_ANALYST — Mon, 11 Aug 2025 09:43:37 GMT

@dbuser24 perhaps a way to determine this would be trying this in the Databricks Free Edition? This could provide some insight.

By the sounds of the error message, it seems more like a permissioning issue with regard to be able to create/read from a directory.

If you have a Free Edition, could you try one of the following notebooks/guides (they've worked for me previously)

I imported this notebook from this resource: https://docs.databricks.com/aws/en/mlflow/end-to-end-example If you look at the navigation bar on the left hand side of the website, you'll see there's a few out of the box examples you can just import into your environment.

I tried this one: https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html and it's worked perfectly fine in my databricks free environment. I'm not too familiar with all the ML components in Databricks but it's working. All it required was for me to create a Catalog called "main". I didn't want to change all the catalog names in the paths that get referenced haha!

Definitely a really cool resource that I've linked though. I'll be learning ML by examples through that. For what it's worth, below is a picture of the ML Notebook working in my environment:

All the best,
BS

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

szymon_dybczak — Mon, 11 Aug 2025 10:05:45 GMT

Hi @dbuser24 ,

What type of cluster do you use? Which access mode and runtime?

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 10:19:01 GMT

Hi @szymon_dybczak - I am using the 16.1 ML cluster

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 10:20:14 GMT

Thanks @BS_THE_ANALYST - Let me try using the above references, and get back on this. Much appreciated !!

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 10:34:25 GMT

@BS_THE_ANALYST I imported the notebook - https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html

still, getting the same error - RESOURCE DOES NOT EXISTS at the step :

with mlflow.start_run() as run:

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

szymon_dybczak — Mon, 11 Aug 2025 11:52:39 GMT

I've tried run your code on my sandbox environment and I didn't encounter any issues.

I did following steps:

- (1) In my workspace under my username directory I've created ML folder:

- (2) Next, I went to my target folder (in this case I've create ML directory) and clicked create MfFLow experiment

- (3) Now, I typed in my expirement name. If you leave artificat location, by default databricks will use following one dbfs:/databricks/mlflow-tracking/<experiment-id>

- (4) Now, I created a new notebook and run the code:

Could you check if you did the same kind of steps?

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 11:49:40 GMT

sure @szymon_dybczak

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 12:32:02 GMT

Thanks @szymon_dybczak for the detailed steps. In addition to the above I had to write the logic to create directory if not present to get it working -

parent_dir = os.path.dirname(experiment_name)

dbutils.fs.mkdirs(parent_dir)

To summarise -
1. Created the ML experiment from within the directory.
2. Verified and corrected the full path of the experiment.

3. Added permission at the experiment level.

4. Added logic to create the directory if not present.

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

dbuser24 — Mon, 11 Aug 2025 12:33:30 GMT

Thanks @szymon_dybczak and @BS_THE_ANALYST - This is resolved, appreciate all the support.

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

szymon_dybczak — Mon, 11 Aug 2025 12:38:45 GMT

Great, glad that we were able to help!

Re: ML experiment giving error - RESOURCE_DOES_NOT_EXIST

BS_THE_ANALYST — Mon, 11 Aug 2025 12:45:01 GMT

can you mark your own post as a solution as well @dbuser24? (would be useful for the additional steps)

Appreciate you feeding back your findings.Congrats on getting it working.

All the best,
BS