topic Re: How to use Databricks secrets on MLFlow conda dependencies? in Machine Learning

How to use Databricks secrets on MLFlow conda dependencies?

sergio-calderon — Thu, 25 Jan 2024 16:55:31 GMT

Hi!

Do you know if it's correct to use the plain user and token for installing a custom dependency (an internal python package) in a mlflow registered model? (it's the only way I get it working because if not it can't install the dependency) It works, but I don't know if this is something dangerous or insecure because I usually use the databricks secrets like this:

%pip install git+https://{dbutils.secrets.get(scope="<my-scope>", key="<user>")}:{dbutils.secrets.get(scope="<scope>", key="<scope>")}@dev.azure.com/<company>/<git-package-project>@<version>

so my conda.yaml for the mlflow is with the plain values:

channels: - conda-forge dependencies: - python=<version> - pip==22.3.1 - pip: - mlflow==2.8.1 - git+https://<plain value of the user>:<plain value of the token>@dev.azure.com/<company>/<git-package-project>@<version> name: mlflow-env

note that in my yaml the values are the output of `dbutils.secrets.get(scope="<my-scope>", key="<user>")` and the other one

Re: How to use Databricks secrets on MLFlow conda dependencies?

Walter_C — Sat, 27 Jan 2024 14:29:59 GMT

While it's possible to use plain user and token for installing a custom dependency in an MLflow registered model, it's not recommended due to security reasons. Instead, Databricks suggests using Databricks secrets to securely store and reference secrets in notebooks or jobs. However, Databricks secrets are not directly accessible in the conda.yaml file, so a workaround such as creating a script to generate the conda.yaml file using the secrets might be necessary. Here's an example of a Python script that can be used to generate the conda.yaml file:

python
user = dbutils.secrets.get(scope="<my-scope>", key="<user>")
token = dbutils.secrets.get(scope="<scope>", key="<scope>")
company = "<company>"
git_package_project = "<git-package-project>"
version = "<version>"

conda_yaml_content = f"""
channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://{user}:{token}@dev.azure.com/{company}/{git_package_project}@{version}
name: mlflow-env
"""

with open("conda.yaml", "w") as file:
    file.write(conda_yaml_content)

Replace the placeholders with your actual values.

Re: How to use Databricks secrets on MLFlow conda dependencies?

sergio-calderon — Tue, 06 Feb 2024 11:26:42 GMT

Thank you very much for the response but using that way is the same as the plain text approach, right? I mean, you are writing it from a Notebook and that's what I've done, but if you open the .yaml file the values are there with plain text.

Re: How to use Databricks secrets on MLFlow conda dependencies?

Froffri — Tue, 12 May 2026 15:47:32 GMT

Did someone solve this? I'm currently forced to download some libraries using an artifactory pypi mirror. However, I wouldn't want to have my secrets pasted in the conda.yaml file as plain text.