cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Creating job under the all-purpose cluster type policy always failed

guangyi
Contributor II

Here is the policy I just created:

 

{
    "node_type_id": {
        "defaultValue": "Standard_D8s_v3",
        "type": "allowlist",
        "values": [
            "Standard_D8s_v3",
            "Standard_D16s_v3"
        ]
    },
    "num_workers": {
        "hidden": false,
        "type": "fixed",
        "value": 1
    },
    "cluster_type": {
        "hidden": false,
        "type": "fixed",
        "value": "all-purpose"
    },
    "azure_attributes.availability": {
        "type": "fixed",
        "value": "SPOT_WITH_FALLBACK_AZURE",
        "hidden": true
    }
}

 

 

Notice a I make the cluster_type value fixed as “all-purpose”. After the policy been created, I continue try to create a job under this policy 

 

 

# The main job for asset_bundle_workflow_demo.

resources:
  jobs:
    asset_bundle_workflow_demo_job:
      name: asset_bundle_workflow_demo_job
      schedule:
        
        # Run every day at 8:37 AM
        quartz_cron_expression: '44 37 8 * * ?'
        timezone_id: Europe/Amsterdam

      tasks:
        - task_key: notebook_task
          job_cluster_key: my_job_cluster
          notebook_task:
            notebook_path: ../src/notebook.ipynb

      job_clusters:
        - job_cluster_key: my_job_cluster
          new_cluster:
            policy_id: xxxxxxxxxxxxxxx
            num_workers: 1
            spark_version: 15.4.x-scala2.12
            node_type_id: Standard_D8s_v3
            cluster_type: all-purpose
            azure_attributes:
              availability: SPOT_WITH_FALLBACK_AZURE

 

And here the error message I got whe try to deploy the asset bundle:

 

Warning: unknown field: cluster_type

  at resources.jobs.asset_bundle_workflow_demo_job.job_clusters[0].new_cluster

  in resources/asset_bundle_workflow_demo_job.yml:35:13

Error: terraform apply: exit status 1

 

Error: cannot create job: Cluster validation error: Validation failed for cluster_type, the value must be all-purpose (is "job")

  with databricks_job.asset_bundle_workflow_demo_job,

  on bundle.tf.json line 63, in resource.databricks_job.asset_bundle_workflow_demo_job:

  63:       }

I’m confused because I just set the cluster_type value in the new_cluster section. What is the wrong with my code?






1 REPLY 1

szymon_dybczak
Contributor III

Hi @guangyi ,

This may be related to the fact that DAB does not support this type of clusters. Unfortunately, this is not very well documented but look at below thread. This feature is requested and it should be available in the future:

Creating All Purpose Cluster in Data Asset Bundles - Databricks Community - 62035

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group