cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to handle configuration for different environment (e.g. DEV, PROD)?

EricOX
New Contributor

May I know any suggested way to handle different environment variables for the same code base? For example, the mount point of Data Lake for DEV, UAT, and PROD. Any recommendations or best practices? Moreover, how to handle Azure DevOps?

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

@Eric Yeungโ€‹ , you can put all your configuration parameters in a file (JSON, CONF, YAML whatever you like) and read that file at the beginning of each program.

I like to use the ConfigFactory in Scala for example.

You only have to make sure the file can be read (f.e. if you put in on your data lake, but the file contains the path to the data lake, you are in trouble).

How to handle devops? That is not an easy one. One can go from as simple as using databricks repos to a fully automated deployment pipeline with automated tests etc.

Your question is perhaps a tad too general to answer.

The databricks docs have some information on CI/CD (if that is what you mean by Azure Devops).

Besides all that: if you use notebooks, use the Repos functionality in databricks.

View solution in original post

1 REPLY 1

-werners-
Esteemed Contributor III

@Eric Yeungโ€‹ , you can put all your configuration parameters in a file (JSON, CONF, YAML whatever you like) and read that file at the beginning of each program.

I like to use the ConfigFactory in Scala for example.

You only have to make sure the file can be read (f.e. if you put in on your data lake, but the file contains the path to the data lake, you are in trouble).

How to handle devops? That is not an easy one. One can go from as simple as using databricks repos to a fully automated deployment pipeline with automated tests etc.

Your question is perhaps a tad too general to answer.

The databricks docs have some information on CI/CD (if that is what you mean by Azure Devops).

Besides all that: if you use notebooks, use the Repos functionality in databricks.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group