topic Re: Passing parameters from DevOps Pipeline/release to DataBricks Notebook in Data Engineering

Passing parameters from DevOps Pipeline/release to DataBricks Notebook

SEOCO — Wed, 03 Nov 2021 11:01:47 GMT

Hi,

This is all a bit new to me.

Does anybody have any idea how to pass a parameter to the Databricks notebook.

I have a DevOps pipeline/release that moves my databricks notebooks towards QA and Production environment. The only problem I am facing is that the DataLake storage location on DEV, QA and PROD are different.

any idea how pass the right datalake storage string/parameter?

Thx

Re: Passing parameters from DevOps Pipeline/release to DataBricks Notebook

Hubert-Dudek — Wed, 03 Nov 2021 12:18:50 GMT

I use separate cluster for dev and production so this way is easy - just use cluster Environment Variables for DataLake storage location:

Re: Passing parameters from DevOps Pipeline/release to DataBricks Notebook

SEOCO — Mon, 08 Nov 2021 12:47:32 GMT

than you for your feedback. I moved it to the KeyVault and it is working.

Re: Passing parameters from DevOps Pipeline/release to DataBricks Notebook

Anonymous — Mon, 08 Nov 2021 16:35:09 GMT

@Mario Walle - If @Hubert Dudek's answer solved the issue, would you be happy to mark his answer as best so that it will be more visible to other members?