Thank you LR, we tried this approach and it worked but for this we would need to define the variable at multiple places, soch as the databricks.yml, the jobs file and the main code.

So we decided to put all these variables in a single config file which is read by the python file and it would replace the values in the sql with values from config file.

We tried using sql parameters but didnt work.