How to use Databricks secrets on MLFlow conda dependencies?

[sergio-calderon]

Hi!

Do you know if it's correct to use the plain user and token for installing a custom dependency (an internal python package) in a mlflow registered model? (it's the only way I get it working because if not it can't install the dependency) It works, but I don't know if this is something dangerous or insecure because I usually use the databricks secrets like this:

%pip install git+https://{dbutils.secrets.get(scope="<my-scope>", key="<user>")}:{dbutils.secrets.get(scope="<scope>", key="<scope>")}@dev.azure.com/<company>/<git-package-project>@<version>

so my conda.yaml for the mlflow is with the plain values:

channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://<plain value of the user>:<plain value of the token>@dev.azure.com/<company>/<git-package-project>@<version>
name: mlflow-env

note that in my yaml the values are the output of `dbutils.secrets.get(scope="<my-scope>", key="<user>")` and the other one

[Walter_C]

While it's possible to use plain user and token for installing a custom dependency in an MLflow registered model, it's not recommended due to security reasons. Instead, Databricks suggests using Databricks secrets to securely store and reference secrets in notebooks or jobs. However, Databricks secrets are not directly accessible in the conda.yaml file, so a workaround such as creating a script to generate the conda.yaml file using the secrets might be necessary. Here's an example of a Python script that can be used to generate the conda.yaml file:

python
user = dbutils.secrets.get(scope="<my-scope>", key="<user>")
token = dbutils.secrets.get(scope="<scope>", key="<scope>")
company = "<company>"
git_package_project = "<git-package-project>"
version = "<version>"

conda_yaml_content = f"""
channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://{user}:{token}@dev.azure.com/{company}/{git_package_project}@{version}
name: mlflow-env
"""

with open("conda.yaml", "w") as file:
    file.write(conda_yaml_content)

Replace the placeholders with your actual values.

[sergio-calderon]

Thank you very much for the response but using that way is the same as the plain text approach, right? I mean, you are writing it from a Notebook and that's what I've done, but if you open the .yaml file the values are there with plain text.

[Froffri]

Did someone solve this? I'm currently forced to download some libraries using an artifactory pypi mirror. However, I wouldn't want to have my secrets pasted in the conda.yaml file as plain text.

[iyashk-DB]

The dynamic conda.yaml generation trick doesn't actually fix this, and you already found that out yourself. Whatever ends up in the final logged conda.yaml is what gets read at environment build time, so if a plaintext credential has to be in that file for pip to authenticate, it's in there for good, no matter how it got written.

The way to actually avoid this is to stop putting credentials in the model's conda.yaml at all, and instead configure a workspace-level default package repository. Under Settings, Compute tab, there's a "Default Package Repositories" section where an admin can set an index-url or extra-index-url pointing at your private repo, and the credentials for that live in a Databricks secret scope on the admin side, not in any individual model's dependency file. Once that's set up, your model's conda.yaml just needs mypackage==1.2.3, no username, no token, no full URL. Databricks documents that Model Serving picks up this workspace-level repo config automatically when it builds the model's environment, so you're not doing anything special per model, it just resolves against the authenticated index behind the scenes.

For Froffri's case with an Artifactory PyPI mirror, that's a clean fit since Artifactory is already speaking the PyPI index protocol, you just point the workspace default repo at it.

For the original git+https install off Azure DevOps though, that one's trickier, because the workspace default repo setting is documented for pip index-style repositories, not raw git URLs. There isn't a supported way to strip credentials out of a git+https://user:token@... line the way there is for an index URL. The cleanest path there is to stop installing directly from the git repo and instead publish the internal package to a proper package feed, Azure Artifacts supports a PyPI-format feed for exactly this, and then treat it the same way as the Artifactory case above. That gets you back to a plain mypackage==1.2.3 line with zero embedded secrets.

One dead end worth mentioning so you don't burn time on it: pip does support ${VAR} environment variable expansion in requirements files for index URLs, so in theory you could write --extra-index-url https://${USER}:${TOKEN}@repo/simple/ and keep the literal token out of the file. But Databricks Model Serving's secret-backed environment variables are only injected at inference runtime, after the environment is already built and dependencies are already installed, so there's no env variable available yet at the point pip actually needs it. That trick works for a local pip install but not for how Model Serving builds the container.