VicS
Databricks Partner

‎11-22-2024 04:54 AM - edited ‎11-22-2024 04:56 AM

It took me a while to realize the distinction of the keys inside the task - so for anyone else looking into this: only one of the following keys can exist in a task definition: 

 

 

      tasks:
        - task_key: ingestion_delta
          # existing_cluster_id: ${var.existing_cluster_id}       # All-purpose cluster
          job_cluster_key: job_cluster                            # Job cluster
          #environment_key: my_environment                        # serverless compute

 

 

Confusing for me is also the fact, that the dependencies / libraries are listed under different keys and in different locations:
- For the serverless, you need to use the key "dependencies" in the "environment" object.
- For the all-purpose and the job-cluster, you need to use the key "libraries" in the "task_key" object.
 
So for a job-cluster, the full, working, example looks like this: 

 

 

resources:
  jobs:
    customer_crm:
      name: customer_crm

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_D4ds_v5
            autoscale:
                min_workers: 1
                max_workers: 4
            custom_tags:
               some_tag: "my_tag"
            apply_policy_default_values: False
            data_security_mode: USER_ISOLATION  # SINGLE_USER (="Single User"), USER_ISOLATION (="Shared"), NONE (="No isolation shared")
            init_scripts:
              - workspace:
                  destination: "/Shared/my_init_script.sh"
            spark_env_vars:
              "PYPI_TOKEN": "{{secrets/<scope-name>/<secret-name>}}"

      tasks:
        - task_key: ingestion_delta
          # existing_cluster_id: ${var.existing_cluster_id}       # All-purpose cluster
          job_cluster_key: job_cluster                            # Job cluster
          #environment_key: my_environment                        # serverless compute
          python_wheel_task:
            package_name: customer_crm
            entry_point: customer_crm_ep_argparse
            named_parameters: { application-name : "myapp" }
          libraries:  # The order of the packages is important (!)
            - pypi:
                package: another-package==1.0.0
                repo: https://pkgs.dev.azure.com/xx/xxx/_packaging/xxx/pypi/simple/
            - whl: ./dist/*.whl