cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use Databricks secrets on MLFlow conda dependencies?

sergio-calderon
New Contributor II

Hi!

Do you know if it's correct to use the plain user and token for installing a custom dependency (an internal python package) in a mlflow registered model? (it's the only way I get it working because if not it can't install the dependency) It works, but I don't know if this is something dangerous or insecure because I usually use the databricks secrets like this:

%pip install git+https://{dbutils.secrets.get(scope="<my-scope>", key="<user>")}:{dbutils.secrets.get(scope="<scope>", key="<scope>")}@dev.azure.com/<company>/<git-package-project>@<version>

so my conda.yaml for the mlflow is with the plain values:

channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://<plain value of the user>:<plain value of the token>@dev.azure.com/<company>/<git-package-project>@<version>
name: mlflow-env

note that in my yaml the values are the output of `dbutils.secrets.get(scope="<my-scope>", key="<user>")` and the other one

3 REPLIES 3

Walter_C
Honored Contributor
Honored Contributor

While it's possible to use plain user and token for installing a custom dependency in an MLflow registered model, it's not recommended due to security reasons. Instead, Databricks suggests using Databricks secrets to securely store and reference secrets in notebooks or jobs. However, Databricks secrets are not directly accessible in the conda.yaml file, so a workaround such as creating a script to generate the conda.yaml file using the secrets might be necessary. Here's an example of a Python script that can be used to generate the conda.yaml file:

python
user = dbutils.secrets.get(scope="<my-scope>", key="<user>")
token = dbutils.secrets.get(scope="<scope>", key="<scope>")
company = "<company>"
git_package_project = "<git-package-project>"
version = "<version>"

conda_yaml_content = f"""
channels:
- conda-forge
dependencies:
- python=<version>
- pip==22.3.1
- pip:
- mlflow==2.8.1
- git+https://{user}:{token}@dev.azure.com/{company}/{git_package_project}@{version}
name: mlflow-env
"""

with open("conda.yaml", "w") as file:
   file.write(conda_yaml_content)

Replace the placeholders with your actual values.

Kaniz_Fatma
Community Manager
Community Manager

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀

 

sergio-calderon
New Contributor II

Thank you very much for the response but using that way is the same as the plain text approach, right? I mean, you are writing it from a Notebook and that's what I've done, but if you open the .yaml file the values are there with plain text.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!