agents.deploy NOT_FOUND: The directory being accessed is not found. error

[stevomcnevo007]

I keep getting the following error although the model definitely does exist and version names and model name is correct

RestException: NOT_FOUND: The directory being accessed is not found.

when calling

# Deploy the model to the review app and a model serving endpoint

agents.deploy(model_name, model_version, scale_to_zero=True)

( i'm on a free trial )

[Alberto_Umana]

The error message "RestException: NOT_FOUND: The directory being accessed is not found" typically indicates that the model or its version cannot be located in the specified directory. Here are a few steps you can take to troubleshoot and resolve this issue:

1. Verify Model URI: Ensure that the model URI is correctly specified. The model URI should follow the format models:/<model_name>/<model_version>. Double-check for any typos in the model name or version.
2. Check Model Registration: Confirm that the model and its version are registered in the MLflow Model Registry. You can do this by navigating to the Model Registry in the Databricks workspace and verifying that the model and version exist.
3. Permissions: Ensure that you have the necessary permissions to access the model and its version. Lack of permissions can sometimes result in a "not found" error.
4. Model Serving Endpoint: If you are using Unity Catalog, ensure that the model serving endpoint is correctly set up and that there are no issues with the endpoint configuration.

Here is an example of how to verify the model URI and check the model registration

import mlflow

# Set the registry URI to Databricks

mlflow.set_registry_uri("databricks-uc")

# Define the model URI

model_uri = f"models:/{model_name}/{model_version}"

# Check if the model exists

try:

    model_info = mlflow.models.get_model_info(model_uri)

    print(f"Model {model_name} version {model_version} exists.")

except mlflow.exceptions.MlflowException as e:

    print(f"Error: {e}")

[stevomcnevo007]

When running this, getting this which looks like it's going to the default directory, this is not where the model sits I think:

Error: RESOURCE_DOES_NOT_EXIST: Routine or Model 'viable_test.default.egg_chat' does not exist.

[Alberto_Umana]

Check the code or configuration files where the model is being referenced or loaded. The path or directory of the model is often specified in these files.

[stevomcnevo007]

Checked the config.yml file and it looks like this:

agent_prompt: "Use functions to interact with questions about eggs / henhouse."

llm_endpoint: "databricks-meta-llama-3-3-70b-instruct"

warehouse_id: "1c7bf12e78b673de"

uc_functions: "main.egg_shop.*"

Still struggling to get this operational

[VZLA]

Hi @stevomcnevo007 

Thanks for sharing the details so far! To help construct the correct resource path and resolve the issue, could you provide a bit more information?

1. Model Details:

   • What is the exact model name and version you’re trying to deploy?
   • Is the model registered under Unity Catalog or the workspace's default model registry?

2. Cluster/Endpoint Setup:

   • Are you using a specific cluster mode (e.g., serverless, shared, or single-user)?
   • Is the llm_endpoint (databricks-meta-llama-3-3-70b-instruct) accessible and correctly set up for your workspace?

3. Error Message Context:

   • Can you confirm where the deployment is failing (e.g., during agents.deploy or a related step)?
   • Share the full error message (or stack trace) if possible, as it can provide more insight.

4. Config File Alignment:

   • Does the config.yml align with the actual model location in the registry? For example, is main.egg_shop.* referencing the correct schema for viable_test.default.egg_chat?

Ultimately, what is the version of this test code with values if it can be shared, to help us diagnose this problem. If the uc_functions parameter (main.egg_shop.*) relates to Unity Catalog, ensure the model's schema (viable_test.default) matches the catalog/schema setup. Also, check you have the necessary permissions to access the model (GRANT USE on Catalog / GRANT SELECT ON Model). 

[stevomcnevo007]

Just wanted to add some clarity on this issue that might shine some light regarding the issue, i've tried setting up multiple models including the Databricks 'dbdemos', 'llm-tag-chatbot' examples and it's resulted in the same directory issues around agents.deploy() 

1. Model Details:

   • What is the exact model name and version you’re trying to deploy?
     - The model is called egg_chat and version 12 at the moment but i've been trying the right version throughout
   • Is the model registered under Unity Catalog or the workspace's default model registry?
     - The model is registered in the Unity Catalog 'My organization' under the 'main' folder, I've tried switching the workspace default catalog to main but the problem still persists  

2. Cluster/Endpoint Setup:

   • Are you using a specific cluster mode (e.g., serverless, shared, or single-user)?
     - Workspace is setup using 'Default Compute and storage' advanced option when creating workspace 
   • Is the llm_endpoint (databricks-meta-llama-3-3-70b-instruct) accessible and correctly set up for your workspace?
     - Yes, everything is working fine when using Playground

3. Error Message Context:

   • Can you confirm where the deployment is failing (e.g., during agents.deploy or a related step)?
     - failing at agents.deploy() due to directory access issue
   • Share the full error message (or stack trace) if possible, as it can provide more insight.
     
     - This is the error message from the dbdemos example but it's the same error for the egg_chat notebook:
     
     RestException: NOT_FOUND: The directory being accessed is not found.
     File <command-2153827424805746>, line 5 2 print(MODEL_NAME_FQN) 3 # Deploy to enable the Review APP and create an API endpoint 4 # Note: scaling down to zero will provide unexpected behavior for the chat app. Set it to false for a prod-ready application. ----> 5 deployment_info = agents.deploy("rag_chatbot.basic_rag_demo", model_version=uc_registered_model_info.version, scale_to_zero=True) 7 instructions_to_reviewer = f"""## Instructions for Testing the Databricks Documentation Assistant chatbot 8 9 Your inputs are invaluable for the development team. By providing detailed feedback and corrections, you help us fix issues and improve the overall quality of the application. We rely on your expertise to identify any gaps or areas needing enhancement.""" 11 # Add the user-facing instructions to the Review App

4. Config File Alignment:

   • Does the config.yml align with the actual model location in the registry? For example, is main.egg_shop.* referencing the correct schema for viable_test.default.egg_chat?
     - viable_test.default.egg_chat does not exist, this is the issue I believe

[VZLA]

- viable_test.default.egg_chat does not exist, this is the issue I believe

I see. Let's ignore the "rag_chatbot.basic_rag_demo" error for now, and focus on the egg_chat.

 So, you stated in your details: "The model is registered in the Unity Catalog 'My organization' under the 'main' folder, I've tried switching the workspace default catalog to main but the problem still persists.". This indicates that the main catalog is where the model resides. Your config.yml includes: uc_functions: "main.egg_shop.*". This suggests that egg_shop is the schema intended for use with this model. The model name egg_chat is consistent throughout your responses, therefore assuming it’s the correct name, the fully qualified name (FQN) should logically be main.egg_shop.egg_chat (please correct me if I'm wrong), instead of viable_test. Viable_test may be getting prepended as default catalog perhaps? Not really sure, but it does sounds like we have to manually set it to be "main" instead.

Can you please run this code below once, from a notebook? It should check both UC and WMR, and ideally find and extract where it is registered if so along with metadata related to it, hope it helps:

from mlflow.tracking.client import MlflowClient
import mlflow

# Initialize MLflow client
client = MlflowClient()

# Specify the model's logical name or partial name to search for
search_model = "egg_chat"

# Function to search for models in a specific registry
def search_registry(uri, model_name):
    mlflow.set_registry_uri(uri)
    client = MlflowClient()
    models = client.search_registered_models()
    for model in models:
        if model_name in model.name:
            return model
    return None

# Check Unity Catalog first
mlflow.set_registry_uri("databricks-uc")
uc_model = search_registry("databricks-uc", search_model)

# If not found in UC, check Workspace Model Registry
if not uc_model:
    print("Model not found in Unity Catalog. Searching Workspace Model Registry...")
    workspace_model = search_registry(None, search_model)
else:
    workspace_model = None

# Function to print detailed model metadata
def print_model_details(model, registry_type):
    print(f"Model found in {registry_type}: {model.name}")
    parts = model.name.split(".")
    if len(parts) == 3:
        catalog, schema, model_name = parts
        print(f"Catalog: {catalog}, Schema: {schema}, Model: {model_name}")
    else:
        print(f"Model Name: {model.name}")
    
    print(f"Tags: {model.tags}")
    if model.latest_versions:
        for version in model.latest_versions:
            print(f" - Version: {version.version}")
            print(f"   Source: {version.source}")
            print(f"   Stage: {version.current_stage}")
            print(f"   Run ID: {version.run_id}")
    else:
        print("No versions available for this model.")

# Print results for both registries
if uc_model:
    print_model_details(uc_model, "Unity Catalog")
if workspace_model:
    print_model_details(workspace_model, "Workspace Model Registry")
if not uc_model and not workspace_model:
    print("Model not found in either Unity Catalog or Workspace Model Registry.")

Then update both the config.yml and agents.deploy as required.

Going back to the original code, what's the output you're getting from:

import mlflow
mlflow.set_registry_uri("databricks-uc")
model_uri = f"models:/main.egg_shop.egg_chat/1"
try:
    model_info = mlflow.models.get_model_info(model_uri)
    print(f"Model {model_name} version {model_version} exists.")
except mlflow.exceptions.MlflowException as e:
    print(f"Error: {e}")​

* Note: you may need to later on check on the model grants, or version, these other side issues should be straight forward to resolve.

[stevomcnevo007]

When running the first section of code this is the reponse:

Model found in Unity Catalog: main.egg_shop.egg_chat Catalog: main, Schema: egg_shop, Model: egg_chat Tags: <bound method RegisteredModelSearch.tags of <RegisteredModelSearch: >> No versions available for this model.

When running second section

Error: NOT_FOUND: The directory being accessed is not found.

[ezermoysis]

Hi @VZLA @Alberto_Umana , I am currently facing the same issue as @stevomcnevo007 . I get exactly the same outputs in the scripts that you have provided above and still unable to deploy the model. Can you please help?

[Alberto_Umana]

Hi @ezermoysis / @stevomcnevo007,

Can you confirm that there are versions of the model registered yet?

This can happen if the model has been registered but no versions have been created or logged yet

[ezermoysis]

Thanks @Alberto_Umana. I think so, yes.

I am using the following code to register the models and when I print the

# Use Unity Catalog as the model registry

mlflow.set_registry_uri("databricks-uc")

print(model_info.model_uri)

# Register the model to the Unity Catalog

uc_registered_model_info = mlflow.register_model(

model_uri=model_info.model_uri, name=UC_MODEL_NAME

)

and when I print

print(uc_registered_model_info)

I get this, so I believe there are versions registered

<ModelVersion: aliases=[], creation_timestamp=1736436606155, current_stage=None, description='', last_updated_timestamp=1736436613763, name='emoysis_catalog.cookbook.sop_chatbot', run_id='', run_link=None, source='dbfs:/databricks/mlflow-tracking/3498969875739103/0ee0f2039ef54180b474e285380fabca/artifacts/agent', status='READY', status_message='', tags={}, user_id='e.moysis@instadeep.com', version='10'>

[ezermoysis]

I note that I am using free version and trying to follow this guide: https://ai-cookbook.io/nbs/5-hands-on-build-poc.html

[ezermoysis]

@Alberto_UmanaBelow I add all the log:

File <command-3498969875739223>, line 12
     10 print(UC_MODEL_NAME)
     11 print(uc_registered_model_info.version)
---> 12 deployment_info = agents.deploy("emoysis_catalog.cookbook.sop_chatbot", 13)
     14 # Wait for the Review App to be ready
     15 print("\nWaiting for endpoint to deploy.  This can take 15 - 20 minutes.", end="")
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/deployments.py:479, in deploy(model_name, model_version, scale_to_zero, environment_vars, instance_profile_arn, tags, workload_size, endpoint_name, **kwargs)
    451 def deploy(
    452     model_name: str,
    453     model_version: int,
   (...)
    460     **kwargs,
    461 ) -> Deployment:
    462     """
    463     Deploy new version of the agent.
    464 
   (...)
    477         Chain deployment metadata.
    478     """
--> 479     _check_model_is_rag_compatible(model_name, model_version)
    480     _check_model_name(model_name)
    481     if endpoint_name is None:
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/utils/mlflow_utils.py:96, in _check_model_is_rag_compatible(model_name, version)
     95 def _check_model_is_rag_compatible(model_name: str, version: int):
---> 96     input_schema, output_schema = _load_model_schema(model_name, version)
     97     return _check_model_is_rag_compatible_helper(input_schema, output_schema)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/utils/mlflow_utils.py:90, in _load_model_schema(model_name, version)
     88 mlflow.set_registry_uri(get_databricks_uc_registry_uri())
     89 model_uri = f"models:/{model_name}/{str(version)}"
---> 90 model_info = mlflow.models.get_model_info(model_uri)
     91 signature = model_info.signature
     92 return signature.inputs, signature.outputs
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/models/model.py:984, in get_model_info(model_uri)
    981 from mlflow.pyfunc import _download_artifact_from_uri
    983 meta_file_uri = model_uri.rstrip("/") + "/" + MLMODEL_FILE_NAME
--> 984 meta_local_path = _download_artifact_from_uri(artifact_uri=meta_file_uri)
    985 model_meta = Model.load(meta_local_path)
    986 return ModelInfo(
    987     artifact_path=model_meta.artifact_path,
    988     flavors=model_meta.flavors,
   (...)
    997     metadata=model_meta.metadata,
    998 )
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/tracking/artifact_utils.py:111, in _download_artifact_from_uri(artifact_uri, output_path, lineage_header_info)
    108 repo = get_artifact_repository(artifact_uri=root_uri)
    110 if isinstance(repo, ModelsArtifactRepository):
--> 111     return repo.download_artifacts(
    112         artifact_path=artifact_path,
    113         dst_path=output_path,
    114         lineage_header_info=lineage_header_info,
    115     )
    116 return repo.download_artifacts(artifact_path=artifact_path, dst_path=output_path)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/models_artifact_repo.py:201, in ModelsArtifactRepository.download_artifacts(self, artifact_path, dst_path, lineage_header_info)
    199 # Pass lineage header info if model is registered in UC
    200 if isinstance(self.repo, UnityCatalogModelsArtifactRepository):
--> 201     model_path = self.repo.download_artifacts(
    202         artifact_path, dst_path, lineage_header_info=lineage_header_info
    203     )
    204 else:
    205     model_path = self.repo.download_artifacts(artifact_path, dst_path)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/unity_catalog_models_artifact_repo.py:157, in UnityCatalogModelsArtifactRepository.download_artifacts(self, artifact_path, dst_path, lineage_header_info)
    156 def download_artifacts(self, artifact_path, dst_path=None, lineage_header_info=None):
--> 157     return self._get_artifact_repo(lineage_header_info=lineage_header_info).download_artifacts(
    158         artifact_path, dst_path
    159     )
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py:268, in ArtifactRepository.download_artifacts(self, artifact_path, dst_path)
    266 # Submit download tasks
    267 futures = {}
--> 268 if self._is_directory(artifact_path):
    269     for file_info in self._iter_artifacts_recursive(artifact_path):
    270         if file_info.is_dir:  # Empty directory
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py:167, in ArtifactRepository._is_directory(self, artifact_path)
    166 def _is_directory(self, artifact_path):
--> 167     listing = self.list_artifacts(artifact_path)
    168     return len(listing) > 0
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/presigned_url_artifact_repo.py:104, in PresignedUrlArtifactRepository.list_artifacts(self, path)
    101 req_body = json.dumps({"page_token": page_token}) if page_token else None
    103 response_proto = ListDirectoryResponse()
--> 104 resp = call_endpoint(
    105     host_creds=self.db_creds,
    106     endpoint=endpoint,
    107     method="GET",
    108     json_body=req_body,
    109     response_proto=response_proto,
    110 )
    111 for dir_entry in resp.contents:
    112     rel_path = posixpath.relpath(dir_entry.path, self.artifact_uri)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/utils/rest_utils.py:370, in call_endpoint(host_creds, endpoint, method, json_body, response_proto, extra_headers)
    367     call_kwargs["json"] = json_body
    368     response = http_request(**call_kwargs)
--> 370 response = verify_rest_response(response, endpoint)
    371 response_to_parse = response.text
    372 js_dict = json.loads(response_to_parse)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/utils/rest_utils.py:240, in verify_rest_response(response, endpoint)
    238 if response.status_code != 200:
    239     if _can_parse_as_json_object(response.text):
--> 240         raise RestException(json.loads(response.text))
    241     else:
    242         base_msg = (
    243             f"API request to endpoint {endpoint} "
    244             f"failed with error code {response.status_code} != 200"
    245         )

[stevomcnevo007]

Any fix here?