How to handle configuration for different environment (e.g. DEV, PROD)?

[EricOX]

May I know any suggested way to handle different environment variables for the same code base? For example, the mount point of Data Lake for DEV, UAT, and PROD. Any recommendations or best practices? Moreover, how to handle Azure DevOps?

[-werners- · Accepted solution]

@Eric Yeung​ , you can put all your configuration parameters in a file (JSON, CONF, YAML whatever you like) and read that file at the beginning of each program.

I like to use the ConfigFactory in Scala for example.

You only have to make sure the file can be read (f.e. if you put in on your data lake, but the file contains the path to the data lake, you are in trouble).

How to handle devops? That is not an easy one. One can go from as simple as using databricks repos to a fully automated deployment pipeline with automated tests etc.

Your question is perhaps a tad too general to answer.

The databricks docs have some information on CI/CD (if that is what you mean by Azure Devops).

Besides all that: if you use notebooks, use the Repos functionality in databricks.