I believe this page is the most interesting to clarify your doubts: Substitutions and variables in Databricks Asset Bundles | Databricks Documentation.

I will try to adapt it to your use case, where I guess you are already adding your bundles variables in databricks.yml.

bundle:
  name: databricks_jobs

include:
  - resources/*.yml
  - resources/*/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dev-workspace.azuredatabricks.net
    variables:
      your_variable: value-dev

  prod:
    mode: production
    workspace:
      host: https://prod-workspace.azuredatabricks.net
    variables:
      your_variable: value-prod

variables:
  your_variable:
    description: Description.
    default: default-value

Here you can define all your variables and need to specify them for each target group. Then as you did already, you access them in your resources by `${var.your_variable}`

When using bundles in your CI/CD tool then you have a few options to overwrite those variables:

1. give the variable value in databricks bundle cli command:

databricks bundle validate --var="your_variable=new-value"

• Set environment an variable: 

export BUNDLE_VAR_your_variable=new-value​

• Use .databricks/bundle/<target>/variable-overrides.json file with content:

{"your_variable": "new-value"}

Then you can always create complex variables (a variable with subfields), and method 3 allows you to overwrite them.

Be aware that there is a priority!

If you need to use the 3rd approach in your DevOps pipeline, then make sure to create the file if it doesn't exist, for example with a bash script step:

mkdir -p .databricks/bundle/dev
echo '{ "your_variable": "new-value" }' > .databricks/bundle/dev/variable-overrides.json

To conclude here is my view on variables:

I personally create a bunch of custom variables in my bundle and usually there are some that I won't change at run time but they change based on the target environment. For example catalog, schema and others. These I'll keep in each target definition.

Then I have Azure DevOps libraries (one per target environment) in which I can save safely authentication things (host, client_id, client_secret). These then I manage with DevOps stages and get the one I need based on your deploy or release strategy.

Finally, I have very few variables I want to modify when I use "databricks deploy" so I use method 1) described above and for instance I pass a git_sha for traceability which could be used as a tag for a job or a parameter in your entrypoint.