topic Re: Passing Different Job Parameters or Environment Variables Based on Targets in DABs in Data Engineering

Passing Different Job Parameters or Environment Variables Based on Targets in DABs

azam-io — Mon, 17 Mar 2025 05:24:21 GMT

Hi, I am currently using DABs in my project and have encountered an issue. Specifically, I need to pass different job parameters or environment variables depending on the target and for specific jobs.
Could you please provide guidance on how to approach this, or any best practices for handling this scenario?
Thank You

Re: Passing Different Job Parameters or Environment Variables Based on Targets in DABs

Nivethan_Venkat — Mon, 17 Mar 2025 10:10:05 GMT

Hi @azam-io,

Please find the best practices here for deploying jobs using Asset Bundles configuration: https://docs.databricks.com/aws/en/dev-tools/bundles/settings

If you would like to pass different parameters for your same job in different environments, you can segregate the jobs with respective configurations in respective target folders as below:

dev ## folder containing respective jobs to be deployed in DEV acc ## folder containing respective jobs to be deployed in ACC prd ## folder containing respective jobs to be deployed in PRD bundles.target.yml ## Here you are maintaining configurations for different environment

By having this approach you can pass different parameters to same job in different environments. In your bundles target, hopefully you will be pointing the respective environment variables for each envs.

Please let us know if you find this helpful or additional clarifications required.

Best Regards,
Nivethan V

Re: Passing Different Job Parameters or Environment Variables Based on Targets in DABs

azam-io — Mon, 17 Mar 2025 11:31:30 GMT

Hi @Nivethan_Venkat, thanks for your response, it's quite helpful. However i would like to know if there is any way to define those variables inside the resource file of the job instead of defining them in separate folders. You know like we do inside targets tag.

Regards,
Azam Ali