I'm using Azure Databricks and I'd like to create a project virtual environment, persisted on a shared compute cluster. As the cluster is shared for many projects, it is necessary to have virtual environments if I want to execute code runs from within Databricks repos. This virtual environment should be easily created via a requirements.txt or conda.yaml file, and activated via the magic command %conda activate <env_name> (or a similar fashion) at the start of a notebook. I cannot find any documentation that lists down the steps, but it seems to be doable according to @Suteja Kanuri​ answer (option2) another question thread here.

Thank you.