Hello @drjb1010 , 

This is a known issue with DBR 14.3 where the `virtualenv` environment manager fails because it depends on `pyenv` to install specific Python versions, but `pyenv` is either not installed or not properly configured in the runtime environment.

## Understanding the Problem

The error occurs because when you specify `env_manager="virtualenv"`, MLflow attempts to create an isolated Python environment matching your model's training environment. It tries to use `pyenv` to install Python 3.9.19, but the command fails with exit code 2, indicating that either:

- `pyenv` is not properly installed in DBR 14.3
- The Python version (3.9.19) cannot be installed via pyenv
- Required dependencies for building Python from source are missing

The transition away from `conda` as an environment manager has left `virtualenv` as an option, but it has dependencies that aren't fully satisfied in DBR 14.3.

Recommended Solution

Use `env_manager="local"` instead of `env_manager="virtualenv"`:

```python
model_udf_score = mlflow.pyfunc.spark_udf(
spark,
model_version_uri,
env_manager="local", # Change from "virtualenv" to "local"
params={"predict_method": "predict_score"}
)
```

What This Means

When using `env_manager="local"`:

- The model will use the cluster's existing Python environment
- No isolated environment creation occurs
- Dependencies must already be installed on the cluster
- You lose the environment isolation benefit but gain stability

Ensuring Dependencies Are Met

Since you're using the local environment, make sure your cluster has the required dependencies installed:

Option 1: Install via notebook
```python
%pip install -r /path/to/requirements.txt
```

Option 2: Cluster Libraries
Install the required libraries directly on the cluster through the Databricks UI under cluster configuration.

Option 3: Init Scripts
Create an init script to install dependencies when the cluster starts.

Alternative Approach

If you absolutely need environment isolation, consider:

Pre-installing dependencies: Before loading the model, manually install all required packages that match your model's dependencies using `%pip install`.

Use Model Serving: Instead of using `spark_udf`, deploy your model to a Model Serving endpoint, which handles environment management differently.

Long-term Recommendation

Monitor Databricks release notes for updates to environment management in future DBR versions. The current state suggests that `env_manager="local"` is the most reliable option until Databricks provides better support for isolated environments without conda dependency.

This issue has been reported by multiple users and appears to be a gap in the current DBR 14.3 implementation. Using `env_manager="local"` is the recommended workaround that will allow you to proceed with your inference workload.

Hope this helps, Louis.