topic Re: DAB run in Data Engineering

DAB run

raghu2 — Mon, 11 Mar 2024 20:40:07 GMT

Hello All,

I am running this command : databricks bundle run -t dev dltPpl_job --debug

Bundle name: dltPpl. Bundle was generated using: databricks bundle init --target dev

Error message: Error: exit status 1
Failed to marshal state to json: unsupported attribute "for_each_task"

Additional error messages:

15:27:05 DEBUG Apply pid=3997 mutator=seq mutator=terraform.Load
15:27:07 ERROR Error: exit status 1
Failed to marshal state to json: unsupported attribute "for_each_task" pid=3997 mutator=seq mutator=terraform.Load
15:27:07 ERROR Error: exit status 1

Appreciate any leads on how to run a unit test from a local development command line.

Re: DAB run

prar_shah — Mon, 09 Sep 2024 18:17:56 GMT

Hello, I am facing the same issue, my job is in .yml form, which I copied from UI after creating the for_each_task loop and selecting "Switch to code version (yaml)". Even after copying it as it is, getting the same error, here is the reference yaml:
```

- task_key: my_custom_task
  depends_on:
    - task_key: my_previous_task
  for_each_task:
    inputs: "{{tasks.my_previous_task.values.parameter1}}"
    task:
      task_key: my_custom_task_iteration
      python_wheel_task:
        package_name: custom_package
        entry_point: program_main
        parameters:
          - --config-path
          - .....
          - "{{input}}"
      job_cluster_key: custom_cluster
      libraries:
        - whl: ....
```

Re: DAB run

prar_shah — Mon, 09 Sep 2024 18:19:16 GMT

Update:
Error is:
Error: cannot create job: No task defined for my_custom_task

Re: DAB run

prar_shah — Tue, 10 Sep 2024 07:54:09 GMT

@Retired_mod
I was trying 'pip install --upgrade databricks' before but after upgrading the version with 'brew upgrade databricks it worked.
Thanks for the help!