cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Asset Bundles complex variable for cluster configuration substitute

jonyvp
New Contributor III

Using this page of the DAB docs, I tried to substitute cluster configuration by a variable. That way, I want to predefine different job cluster configurations. Doing exactly what is used in the docs yields this error:


Error: failed to load [...]/specific_job.yml: error unmarshaling JSON: json: cannot unmarshal object into Go struct field Variable.variables.default of type string

1 ACCEPTED SOLUTION

Accepted Solutions

jonyvp
New Contributor III

@filipniziol that's it! Upgrading the CLI ( 0.209.1 -> 0.228.0 ) did the trick.

View solution in original post

7 REPLIES 7

filipniziol
Contributor

Hi @jonyvp ,
could you share your .yml?

jonyvp
New Contributor III

I do an include of the yml in my databricks.yml. The yml pasted below:

variables:
my_cluster:
description: "My cluster definition"
type: complex
default:
spark_version: "13.2.x-scala2.11"
node_type_id: "Standard_DS3_v2"
num_workers: 2
spark_conf:
spark.speculation: true
spark.databricks.delta.retentionDurationCheck.enabled: false

resources:
jobs:
my_job:
name: "my_job"
max_concurrent_runs: 1
job_clusters:
- job_cluster_key: my_cluster_key
new_cluster: ${var.my_cluster}
tasks:
- task_key: my_job
spark_python_task:
python_file: /Workspace/Shared/.bundle/[...]/specific_task.py
job_cluster_key: my_cluster_key
libraries:
- whl: ../dist/*.whl

Hi @jonyvp ,

This won't work. The ${var.my_cluster} can substitute just one individual value.
With variable you can substitute specific fields, but you cannot substitue the entire cluster configuration object at once. Thus the error.

Check the article on how to override cluster settings:
Override cluster settings in Databricks Asset Bundles | Databricks on AWS

jonyvp
New Contributor III

Hi filipniziol, according to this docs, it should be possible?

https://docs.databricks.com/en/dev-tools/bundles/variables.html#define-a-complex-variable

 

filipniziol
Contributor

@jonyvp , wow, that's something new to me! Thanks for sharing!
The error though is about inability to convert your object to string, so it expects a string.
Are you using the newest databricks cli? Maybe that's a relatively new feature and you need to use the up-to-date cli

jonyvp
New Contributor III

@filipniziol that's it! Upgrading the CLI ( 0.209.1 -> 0.228.0 ) did the trick.

filipniziol
Contributor

Amazing! Great!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group