How to use Databricks secrets on MLFlow conda dependencies?

[sergio-calderon]

Hi!

Do you know if it's correct to use the plain user and token for installing a custom dependency (an internal python package) in a mlflow registered model? (it's the only way I get it working because if not it can't install the dependency) It works, but I don't know if this is something dangerous or insecure because I usually use the databricks secrets like this:

%pip install git+https://{dbutils.secrets.get(scope="<my-scope>", key="<user>")}:{dbutils.secrets.get(scope="<scope>", key="<scope>")}@dev.azure.com/<company>/<git-package-project>@<version>

so my conda.yaml for the mlflow is with the plain values:

channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://<plain value of the user>:<plain value of the token>@dev.azure.com/<company>/<git-package-project>@<version>
name: mlflow-env

note that in my yaml the values are the output of `dbutils.secrets.get(scope="<my-scope>", key="<user>")` and the other one

[Walter_C]

While it's possible to use plain user and token for installing a custom dependency in an MLflow registered model, it's not recommended due to security reasons. Instead, Databricks suggests using Databricks secrets to securely store and reference secrets in notebooks or jobs. However, Databricks secrets are not directly accessible in the conda.yaml file, so a workaround such as creating a script to generate the conda.yaml file using the secrets might be necessary. Here's an example of a Python script that can be used to generate the conda.yaml file:

python
user = dbutils.secrets.get(scope="<my-scope>", key="<user>")
token = dbutils.secrets.get(scope="<scope>", key="<scope>")
company = "<company>"
git_package_project = "<git-package-project>"
version = "<version>"

conda_yaml_content = f"""
channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://{user}:{token}@dev.azure.com/{company}/{git_package_project}@{version}
name: mlflow-env
"""

with open("conda.yaml", "w") as file:
    file.write(conda_yaml_content)

Replace the placeholders with your actual values.

[Kaniz]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀

 

[sergio-calderon]

Thank you very much for the response but using that way is the same as the plain text approach, right? I mean, you are writing it from a Notebook and that's what I've done, but if you open the .yaml file the values are there with plain text.