topic Re: Complex type variable in Databricks.yml not working in Data Engineering

Complex type variable in Databricks.yml not working

guangyi — Fri, 02 Aug 2024 05:42:56 GMT

For example here I extract the schedule parameter as a complex type variable:

variables: schedule: description: schedule time type: complex default: quartz_cron_expression: '0 22 17 * * ?' timezone_id: Asia/Shanghai pause_status: "UNPAUSED"

And here is how I use in my job definition:

resources: jobs: my_workflow_job: name: …… schedule: ${var.schedule}

When I validate the yaml file via command: `databricks bundle validate -t dev -o json`, it told me:

Error: no value assigned to required variable schedule. Assignment can be done through the "--var" flag or by setting the BUNDLE_VAR_schedule environment variable

I don’t know what’s wrong with my code, I follow the example in the document https://docs.databricks.com/en/dev-tools/bundles/settings.html#define-a-complex-variable to define the complex type variable

Re: Complex type variable in Databricks.yml not working

guangyi — Fri, 02 Aug 2024 08:30:47 GMT

It’s strange, it can pass the validation on my colleague’s laptop. I still cannot pass the validation even restart my Terminal and Macbook

Re: Complex type variable in Databricks.yml not working

pavlosskev — Fri, 02 Aug 2024 09:22:07 GMT

If the validation is fine on your colleague's laptop and not on yours, my first assumption would be that it's a version issue. Do you have the same Databricks CLI version as your colleagues? You can check with

databricks --version

Also according to this documentation, you need Databricks CLI version 0.218.0 or higher to use bundles.

Re: Complex type variable in Databricks.yml not working

guangyi — Fri, 02 Aug 2024 09:25:15 GMT

You are right, I solved the issue by upgrade the Databricks CLI. My local CLI version is 0.221. According to the release note, it looks like they support the complex variable since 0.222 only.