topic Re: Databricks Asset Bundles complex variable for cluster configuration substitute in Data Engineering

Databricks Asset Bundles complex variable for cluster configuration substitute

jonyvp — Wed, 18 Sep 2024 10:16:36 GMT

Using this page of the DAB docs, I tried to substitute cluster configuration by a variable. That way, I want to predefine different job cluster configurations. Doing exactly what is used in the docs yields this error:

Error: failed to load [...]/specific_job.yml: error unmarshaling JSON: json: cannot unmarshal object into Go struct field Variable.variables.default of type string

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

filipniziol — Wed, 18 Sep 2024 11:25:16 GMT

Hi @jonyvp ,
could you share your .yml?

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

jonyvp — Wed, 18 Sep 2024 11:30:31 GMT

I do an include of the yml in my databricks.yml. The yml pasted below:

variables:
  my_cluster:
    description: "My cluster definition"
    type: complex
    default:
      spark_version: "13.2.x-scala2.11"
      node_type_id: "Standard_DS3_v2"
      num_workers: 2
      spark_conf:
        spark.speculation: true
        spark.databricks.delta.retentionDurationCheck.enabled: false

resources:
  jobs:
    my_job:
      name: "my_job"
      max_concurrent_runs: 1
      job_clusters:
        - job_cluster_key: my_cluster_key
          new_cluster: ${var.my_cluster}
      tasks:
        - task_key: my_job
          spark_python_task:
            python_file: /Workspace/Shared/.bundle/[...]/specific_task.py
          job_cluster_key: my_cluster_key
          libraries:
            - whl: ../dist/*.whl

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

filipniziol — Wed, 18 Sep 2024 11:40:15 GMT

Hi @jonyvp ,

This won't work. The ${var.my_cluster} can substitute just one individual value.
With variable you can substitute specific fields, but you cannot substitue the entire cluster configuration object at once. Thus the error.

Check the article on how to override cluster settings:
Override cluster settings in Databricks Asset Bundles | Databricks on AWS

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

jonyvp — Wed, 18 Sep 2024 11:42:00 GMT

Hi filipniziol, according to this docs, it should be possible?

https://docs.databricks.com/en/dev-tools/bundles/variables.html#define-a-complex-variable

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

filipniziol — Wed, 18 Sep 2024 12:07:59 GMT

@jonyvp , wow, that's something new to me! Thanks for sharing!
The error though is about inability to convert your object to string, so it expects a string.
Are you using the newest databricks cli? Maybe that's a relatively new feature and you need to use the up-to-date cli

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

jonyvp — Wed, 18 Sep 2024 12:24:26 GMT

@filipniziol that's it! Upgrading the CLI ( 0.209.1 -> 0.228.0 ) did the trick.

Re: Databricks Asset Bundles complex variable for cluster configuration substitute

filipniziol — Wed, 18 Sep 2024 12:42:14 GMT

Amazing! Great!