topic Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error in Data Engineering

agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Fri, 27 Dec 2024 13:20:30 GMT

I keep getting the following error although the model definitely does exist and version names and model name is correct

RestException: NOT_FOUND: The directory being accessed is not found.

when calling

# Deploy the model to the review app and a model serving endpoint

agents.deploy(model_name, model_version, scale_to_zero=True)

( i'm on a free trial )

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

Alberto_Umana — Fri, 27 Dec 2024 13:43:56 GMT

The error message "RestException: NOT_FOUND: The directory being accessed is not found" typically indicates that the model or its version cannot be located in the specified directory. Here are a few steps you can take to troubleshoot and resolve this issue:

Verify Model URI: Ensure that the model URI is correctly specified. The model URI should follow the format models:/<model_name>/<model_version>. Double-check for any typos in the model name or version.
Check Model Registration: Confirm that the model and its version are registered in the MLflow Model Registry. You can do this by navigating to the Model Registry in the Databricks workspace and verifying that the model and version exist.
Permissions: Ensure that you have the necessary permissions to access the model and its version. Lack of permissions can sometimes result in a "not found" error.
Model Serving Endpoint: If you are using Unity Catalog, ensure that the model serving endpoint is correctly set up and that there are no issues with the endpoint configuration.

Here is an example of how to verify the model URI and check the model registration

import mlflow

# Set the registry URI to Databricks

mlflow.set_registry_uri("databricks-uc")

# Define the model URI

model_uri = f"models:/{model_name}/{model_version}"

# Check if the model exists

try:

model_info = mlflow.models.get_model_info(model_uri)

print(f"Model {model_name} version {model_version} exists.")

except mlflow.exceptions.MlflowException as e:

print(f"Error: {e}")

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Fri, 27 Dec 2024 15:54:05 GMT

When running this, getting this which looks like it's going to the default directory, this is not where the model sits I think:

Error: RESOURCE_DOES_NOT_EXIST: Routine or Model 'viable_test.default.egg_chat' does not exist.

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

Alberto_Umana — Fri, 27 Dec 2024 20:17:38 GMT

Check the code or configuration files where the model is being referenced or loaded. The path or directory of the model is often specified in these files.

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Mon, 30 Dec 2024 13:10:30 GMT

Checked the config.yml file and it looks like this:

agent_prompt: "Use functions to interact with questions about eggs / henhouse."

llm_endpoint: "databricks-meta-llama-3-3-70b-instruct"

warehouse_id: "1c7bf12e78b673de"

uc_functions: "main.egg_shop.*"

Still struggling to get this operational

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

VZLA — Mon, 30 Dec 2024 17:08:34 GMT

Hi @stevomcnevo007

Thanks for sharing the details so far! To help construct the correct resource path and resolve the issue, could you provide a bit more information?

Model Details:
- What is the exact model name and version you’re trying to deploy?
- Is the model registered under Unity Catalog or the workspace's default model registry?
Cluster/Endpoint Setup:
- Are you using a specific cluster mode (e.g., serverless, shared, or single-user)?
- Is the llm_endpoint (databricks-meta-llama-3-3-70b-instruct) accessible and correctly set up for your workspace?
Error Message Context:
- Can you confirm where the deployment is failing (e.g., during agents.deploy or a related step)?
- Share the full error message (or stack trace) if possible, as it can provide more insight.
Config File Alignment:
- Does the config.yml align with the actual model location in the registry? For example, is main.egg_shop.* referencing the correct schema for viable_test.default.egg_chat?

Ultimately, what is the version of this test code with values if it can be shared, to help us diagnose this problem. If the uc_functions parameter (main.egg_shop.*) relates to Unity Catalog, ensure the model's schema (viable_test.default) matches the catalog/schema setup. Also, check you have the necessary permissions to access the model (GRANT USE on Catalog / GRANT SELECT ON Model).

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Tue, 31 Dec 2024 09:33:44 GMT

Just wanted to add some clarity on this issue that might shine some light regarding the issue, i've tried setting up multiple models including the Databricks 'dbdemos', 'llm-tag-chatbot' examples and it's resulted in the same directory issues around agents.deploy()

Model Details:
- What is the exact model name and version you’re trying to deploy?
  - The model is called egg_chat and version 12 at the moment but i've been trying the right version throughout
- Is the model registered under Unity Catalog or the workspace's default model registry?
  - The model is registered in the Unity Catalog 'My organization' under the 'main' folder, I've tried switching the workspace default catalog to main but the problem still persists
Cluster/Endpoint Setup:
- Are you using a specific cluster mode (e.g., serverless, shared, or single-user)?
  - Workspace is setup using 'Default Compute and storage' advanced option when creating workspace
- Is the llm_endpoint (databricks-meta-llama-3-3-70b-instruct) accessible and correctly set up for your workspace?
  - Yes, everything is working fine when using Playground
Error Message Context:
- Can you confirm where the deployment is failing (e.g., during agents.deploy or a related step)?
  - failing at agents.deploy() due to directory access issue
- Share the full error message (or stack trace) if possible, as it can provide more insight.
  - This is the error message from the dbdemos example but it's the same error for the egg_chat notebook:
  
  RestException: NOT_FOUND: The directory being accessed is not found.
  File <command-2153827424805746>, line 5 2 print(MODEL_NAME_FQN) 3 # Deploy to enable the Review APP and create an API endpoint 4 # Note: scaling down to zero will provide unexpected behavior for the chat app. Set it to false for a prod-ready application. ----> 5 deployment_info = agents.deploy("rag_chatbot.basic_rag_demo", model_version=uc_registered_model_info.version, scale_to_zero=True) 7 instructions_to_reviewer = f"""## Instructions for Testing the Databricks Documentation Assistant chatbot 8 9 Your inputs are invaluable for the development team. By providing detailed feedback and corrections, you help us fix issues and improve the overall quality of the application. We rely on your expertise to identify any gaps or areas needing enhancement.""" 11 # Add the user-facing instructions to the Review App
Config File Alignment:
- Does the config.yml align with the actual model location in the registry? For example, is main.egg_shop.* referencing the correct schema for viable_test.default.egg_chat?
  - viable_test.default.egg_chat does not exist, this is the issue I believe

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

VZLA — Tue, 31 Dec 2024 12:34:21 GMT

- viable_test.default.egg_chat does not exist, this is the issue I believe

I see. Let's ignore the "rag_chatbot.basic_rag_demo" error for now, and focus on the egg_chat.

So, you stated in your details: "The model is registered in the Unity Catalog 'My organization' under the 'main' folder, I've tried switching the workspace default catalog to main but the problem still persists.". This indicates that the main catalog is where the model resides. Your config.yml includes: uc_functions: "main.egg_shop.*". This suggests that egg_shop is the schema intended for use with this model. The model name egg_chat is consistent throughout your responses, therefore assuming it’s the correct name, the fully qualified name (FQN) should logically be main.egg_shop.egg_chat (please correct me if I'm wrong), instead of viable_test. Viable_test may be getting prepended as default catalog perhaps? Not really sure, but it does sounds like we have to manually set it to be "main" instead.

Can you please run this code below once, from a notebook? It should check both UC and WMR, and ideally find and extract where it is registered if so along with metadata related to it, hope it helps:

from mlflow.tracking.client import MlflowClient import mlflow # Initialize MLflow client client = MlflowClient() # Specify the model's logical name or partial name to search for search_model = "egg_chat" # Function to search for models in a specific registry def search_registry(uri, model_name): mlflow.set_registry_uri(uri) client = MlflowClient() models = client.search_registered_models() for model in models: if model_name in model.name: return model return None # Check Unity Catalog first mlflow.set_registry_uri("databricks-uc") uc_model = search_registry("databricks-uc", search_model) # If not found in UC, check Workspace Model Registry if not uc_model: print("Model not found in Unity Catalog. Searching Workspace Model Registry...") workspace_model = search_registry(None, search_model) else: workspace_model = None # Function to print detailed model metadata def print_model_details(model, registry_type): print(f"Model found in {registry_type}: {model.name}") parts = model.name.split(".") if len(parts) == 3: catalog, schema, model_name = parts print(f"Catalog: {catalog}, Schema: {schema}, Model: {model_name}") else: print(f"Model Name: {model.name}") print(f"Tags: {model.tags}") if model.latest_versions: for version in model.latest_versions: print(f" - Version: {version.version}") print(f" Source: {version.source}") print(f" Stage: {version.current_stage}") print(f" Run ID: {version.run_id}") else: print("No versions available for this model.") # Print results for both registries if uc_model: print_model_details(uc_model, "Unity Catalog") if workspace_model: print_model_details(workspace_model, "Workspace Model Registry") if not uc_model and not workspace_model: print("Model not found in either Unity Catalog or Workspace Model Registry.")

Then update both the config.yml and agents.deploy as required.

Going back to the original code, what's the output you're getting from:

import mlflow mlflow.set_registry_uri("databricks-uc") model_uri = f"models:/main.egg_shop.egg_chat/1" try: model_info = mlflow.models.get_model_info(model_uri) print(f"Model {model_name} version {model_version} exists.") except mlflow.exceptions.MlflowException as e: print(f"Error: {e}")

* Note: you may need to later on check on the model grants, or version, these other side issues should be straight forward to resolve.

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Tue, 31 Dec 2024 13:36:01 GMT

When running the first section of code this is the reponse:

Model found in Unity Catalog: main.egg_shop.egg_chat Catalog: main, Schema: egg_shop, Model: egg_chat Tags: <bound method RegisteredModelSearch.tags of <RegisteredModelSearch: >> No versions available for this model.

When running second section

Error: NOT_FOUND: The directory being accessed is not found.

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

ezermoysis — Thu, 09 Jan 2025 16:00:52 GMT

Hi @VZLA @Alberto_Umana , I am currently facing the same issue as @stevomcnevo007 . I get exactly the same outputs in the scripts that you have provided above and still unable to deploy the model. Can you please help?

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

Alberto_Umana — Thu, 09 Jan 2025 16:42:25 GMT

Hi @ezermoysis / @stevomcnevo007,

Can you confirm that there are versions of the model registered yet?

This can happen if the model has been registered but no versions have been created or logged yet

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

ezermoysis — Thu, 09 Jan 2025 16:52:06 GMT

Thanks @Alberto_Umana. I think so, yes.

I am using the following code to register the models and when I print the

# Use Unity Catalog as the model registry

mlflow.set_registry_uri("databricks-uc")

print(model_info.model_uri)

# Register the model to the Unity Catalog

uc_registered_model_info = mlflow.register_model(

model_uri=model_info.model_uri, name=UC_MODEL_NAME

)

and when I print

print(uc_registered_model_info)

I get this, so I believe there are versions registered

<ModelVersion: aliases=[], creation_timestamp=1736436606155, current_stage=None, description='', last_updated_timestamp=1736436613763, name='emoysis_catalog.cookbook.sop_chatbot', run_id='', run_link=None, source='dbfs:/databricks/mlflow-tracking/3498969875739103/0ee0f2039ef54180b474e285380fabca/artifacts/agent', status='READY', status_message='', tags={}, user_id='e.moysis@instadeep.com', version='10'>

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

ezermoysis — Thu, 09 Jan 2025 16:56:19 GMT

I note that I am using free version and trying to follow this guide: https://ai-cookbook.io/nbs/5-hands-on-build-poc.html

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

ezermoysis — Fri, 10 Jan 2025 13:38:48 GMT

@Alberto_UmanaBelow I add all the log:

File <command-3498969875739223>, line 12 10 print(UC_MODEL_NAME) 11 print(uc_registered_model_info.version) ---> 12 deployment_info = agents.deploy("emoysis_catalog.cookbook.sop_chatbot", 13) 14 # Wait for the Review App to be ready 15 print("\nWaiting for endpoint to deploy. This can take 15 - 20 minutes.", end="") File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/deployments.py:479, in deploy(model_name, model_version, scale_to_zero, environment_vars, instance_profile_arn, tags, workload_size, endpoint_name, **kwargs) 451 def deploy( 452 model_name: str, 453 model_version: int, (...) 460 **kwargs, 461 ) -> Deployment: 462 """ 463 Deploy new version of the agent. 464 (...) 477 Chain deployment metadata. 478 """ --> 479 _check_model_is_rag_compatible(model_name, model_version) 480 _check_model_name(model_name) 481 if endpoint_name is None: File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/utils/mlflow_utils.py:96, in _check_model_is_rag_compatible(model_name, version) 95 def _check_model_is_rag_compatible(model_name: str, version: int): ---> 96 input_schema, output_schema = _load_model_schema(model_name, version) 97 return _check_model_is_rag_compatible_helper(input_schema, output_schema) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/databricks/agents/utils/mlflow_utils.py:90, in _load_model_schema(model_name, version) 88 mlflow.set_registry_uri(get_databricks_uc_registry_uri()) 89 model_uri = f"models:/{model_name}/{str(version)}" ---> 90 model_info = mlflow.models.get_model_info(model_uri) 91 signature = model_info.signature 92 return signature.inputs, signature.outputs File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/models/model.py:984, in get_model_info(model_uri) 981 from mlflow.pyfunc import _download_artifact_from_uri 983 meta_file_uri = model_uri.rstrip("/") + "/" + MLMODEL_FILE_NAME --> 984 meta_local_path = _download_artifact_from_uri(artifact_uri=meta_file_uri) 985 model_meta = Model.load(meta_local_path) 986 return ModelInfo( 987 artifact_path=model_meta.artifact_path, 988 flavors=model_meta.flavors, (...) 997 metadata=model_meta.metadata, 998 ) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/tracking/artifact_utils.py:111, in _download_artifact_from_uri(artifact_uri, output_path, lineage_header_info) 108 repo = get_artifact_repository(artifact_uri=root_uri) 110 if isinstance(repo, ModelsArtifactRepository): --> 111 return repo.download_artifacts( 112 artifact_path=artifact_path, 113 dst_path=output_path, 114 lineage_header_info=lineage_header_info, 115 ) 116 return repo.download_artifacts(artifact_path=artifact_path, dst_path=output_path) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/models_artifact_repo.py:201, in ModelsArtifactRepository.download_artifacts(self, artifact_path, dst_path, lineage_header_info) 199 # Pass lineage header info if model is registered in UC 200 if isinstance(self.repo, UnityCatalogModelsArtifactRepository): --> 201 model_path = self.repo.download_artifacts( 202 artifact_path, dst_path, lineage_header_info=lineage_header_info 203 ) 204 else: 205 model_path = self.repo.download_artifacts(artifact_path, dst_path) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/unity_catalog_models_artifact_repo.py:157, in UnityCatalogModelsArtifactRepository.download_artifacts(self, artifact_path, dst_path, lineage_header_info) 156 def download_artifacts(self, artifact_path, dst_path=None, lineage_header_info=None): --> 157 return self._get_artifact_repo(lineage_header_info=lineage_header_info).download_artifacts( 158 artifact_path, dst_path 159 ) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py:268, in ArtifactRepository.download_artifacts(self, artifact_path, dst_path) 266 # Submit download tasks 267 futures = {} --> 268 if self._is_directory(artifact_path): 269 for file_info in self._iter_artifacts_recursive(artifact_path): 270 if file_info.is_dir: # Empty directory File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py:167, in ArtifactRepository._is_directory(self, artifact_path) 166 def _is_directory(self, artifact_path): --> 167 listing = self.list_artifacts(artifact_path) 168 return len(listing) > 0 File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/store/artifact/presigned_url_artifact_repo.py:104, in PresignedUrlArtifactRepository.list_artifacts(self, path) 101 req_body = json.dumps({"page_token": page_token}) if page_token else None 103 response_proto = ListDirectoryResponse() --> 104 resp = call_endpoint( 105 host_creds=self.db_creds, 106 endpoint=endpoint, 107 method="GET", 108 json_body=req_body, 109 response_proto=response_proto, 110 ) 111 for dir_entry in resp.contents: 112 rel_path = posixpath.relpath(dir_entry.path, self.artifact_uri) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/utils/rest_utils.py:370, in call_endpoint(host_creds, endpoint, method, json_body, response_proto, extra_headers) 367 call_kwargs["json"] = json_body 368 response = http_request(**call_kwargs) --> 370 response = verify_rest_response(response, endpoint) 371 response_to_parse = response.text 372 js_dict = json.loads(response_to_parse) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-0ea80acf-754f-4b74-9414-f078ba584e3e/lib/python3.10/site-packages/mlflow/utils/rest_utils.py:240, in verify_rest_response(response, endpoint) 238 if response.status_code != 200: 239 if _can_parse_as_json_object(response.text): --> 240 raise RestException(json.loads(response.text)) 241 else: 242 base_msg = ( 243 f"API request to endpoint {endpoint} " 244 f"failed with error code {response.status_code} != 200" 245 )

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

stevomcnevo007 — Mon, 13 Jan 2025 16:47:16 GMT

Any fix here?

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

VZLA — Tue, 14 Jan 2025 13:13:21 GMT

@stevomcnevo007 I would suggest raising a support ticket at this point for direct collaboration on the resolution. Meanwhile I'd like to understand if there are any errors related to permission or artifact paths, or some additional error details to pinpoint the root cause:

import mlflow from databricks import agents # Set Unity Catalog as the registry mlflow.set_registry_uri("databricks-uc") # Verify the registered model and its schema print(uc_registered_model_info) model_info = mlflow.models.get_model_info( f"models:/emoysis_catalog.cookbook.sop_chatbot/{uc_registered_model_info.version}" ) print(model_info.signature) # Deploy the model deployment_info = agents.deploy( model_name="emoysis_catalog.cookbook.sop_chatbot", model_version=int(uc_registered_model_info.version), environment_vars={}, ) print(deployment_info)

Re: agents.deploy NOT_FOUND: The directory being accessed is not found. error

ezermoysis — Thu, 16 Jan 2025 18:18:17 GMT

Does the model need to be served before deployment?