cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Complex type variable in Databricks.yml not working

guangyi
Contributor

For example here I extract the schedule parameter as a complex type variable:

 

variables:
  schedule:
    description: schedule time
    type: complex
    default:
      quartz_cron_expression: '0 22 17 * * ?'
      timezone_id: Asia/Shanghai
      pause_status: "UNPAUSED"

 

And here is how I use in my job definition:

 

resources:
  jobs:
    my_workflow_job:
      name: โ€ฆโ€ฆ
      schedule: ${var.schedule}

 

When I validate the yaml file via command: `databricks bundle validate -t dev -o json`, it told me:

Error: no value assigned to required variable schedule. Assignment can be done through the "--var" flag or by setting the BUNDLE_VAR_schedule environment variable

I donโ€™t know whatโ€™s wrong with my code, I follow the example in the document https://docs.databricks.com/en/dev-tools/bundles/settings.html#define-a-complex-variable to define the complex type variable

1 ACCEPTED SOLUTION

Accepted Solutions

You are right, I solved the issue by upgrade the Databricks CLI. My local CLI version is 0.221. According to the release note, it looks like they support the complex variable since 0.222 only.

View solution in original post

3 REPLIES 3

guangyi
Contributor

Itโ€™s strange, it can pass the validation on my colleagueโ€™s laptop. I still cannot pass the validation even restart my Terminal and Macbook

pavlosskev
New Contributor III

 

If the validation is fine on your colleague's laptop and not on yours, my first assumption would be that it's a version issue. Do you have the same Databricks CLI version as your colleagues? You can check with 

 

databricks --version

 

Also according to this documentation, you need Databricks CLI version 0.218.0 or higher to use bundles.

 

You are right, I solved the issue by upgrade the Databricks CLI. My local CLI version is 0.221. According to the release note, it looks like they support the complex variable since 0.222 only.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group