Databricks Community

AnhPT · 2 weeks ago

Does DAB support any form of task reuse or shared task templates across job definitions — so the wrapper tasks don't need to be physically repeated in every file?
Is there a recommended pattern to externalize the orchestration wrapper (logging, skip logic) from individual job definitions — perhaps as a shared job that other jobs trigger?
For teams using code generation to produce DAB YAML at scale: how do you handle jobs that deviate from the standard template without forking into unmaintained hand-written files?

AnhPT · 2 weeks ago

1.Current State
The deployment uses Dâ Asset Bundle (DAB) with the following structure:

entries/ ← developer-authored business config
---------*.yml ← what tables to read/write, pre-check conditions
resources/ ← dbx Bundle YAML, ready to deploy

Every generated resources/*.yml contains an identical 6-task wrapper pattern:

- task_key: set_params # identical in all
- task_key: create_log # identical in all
- task_key: skip_run # identical in all
- task_key: finish_log_skipped # identical in all
- task_key: finish_log_success # identical in all

← full job definitions consumed by `databricks bundle deploy`
resources/common/ ← shared variables: clusters, tags, notifications
joblib/helper/entry_converter/templates/
------ single_job_resource_template.j2
------ multi_jobs_resource_template.j2

juan_maedo · 2 weeks ago

Hi @AnhPT ,

1 and 2. I’ll combine these two since it might be helpful—I can tell you that I’ve used this approach successfully. In a situation I’ve encountered with similar archetypes—where utilities or steps were exactly the same across bundles—and where, most importantly, if one changes, they all must change (cluster configurations, catalogs, run and manage permissions, etc.)—I’ve used a cross-bundle shared repository to store those utilities or core components used by the rest. This was accessed by defining the relative paths to it. This way, if I update a function or a notebook, when I deploy it to the next environment, every new job will use it. In my case, I deploy it using SPNs in a folder with read-only access for users, while the SPNs have execute permissions to compile it in the jobs—ensuring that “no one” can make manual changes.

3. Without a doubt, the best option for this—and to streamline the process (including agents)—is to use custom templates during initialization, whether from a Git/DevOps tool or directly within the workspaces. https://docs.databricks.com/aws/en/dev-tools/bundles/templates#custom-bundle-templates

AnhPT · 2 weeks ago

i think not good. when i deploy service k8s. i use kustomize yaml. i can using template. Jobs just update value change.

anuj_lathi · Thursday

Short answer: DAB has no native task-level templates or includes. A YAML file can't say "inject these 5 tasks into every job." Reuse has to come from one of three places: generating the YAML (what you do today), Python for bundles, or moving the wrapper out of the job graph.

1. Reuse inside DAB itself

YAML anchors/aliases are standard YAML, so they work, but only within a single file. They don't cross files, so they won't solve your problem.
include and resources/common/ are good for sharing variables, clusters, tags and notifications [1][2]. That's what you already do. They share resource-level config, not a block of tasks that gets spliced into each job.
Python for bundles is the closest thing to real task reuse. Jobs are a supported resource type for Python [2], so you can write a function like with_standard_wrapper(job, business_tasks) that returns the full job. You can also apply it as a mutator across every job in the bundle. The wrapper then lives in one tested Python module instead of N generated files. Check the Python-for-bundles docs for your CLI version, since it's still evolving.

2. Shared wrapper as a job other jobs trigger

This works, but it has tradeoffs, so be deliberate about it.

Lakeflow Jobs supports a Run Job task type that calls another job [3]. A parent "orchestrator" job can run the logging/skip logic and then run the business job. You can reference the child with a bundle variable like ${resources.jobs.<name>.id} rather than hardcoding IDs.
You pay for it with an extra nested run per execution, so lineage, retries, and run-level alerting are split across two jobs. Debugging "why did this skip?" means chasing a parent and a child.
It also inverts your current structure. Today the wrapper sits in the same DAG as the business tasks, so finish_log_success can depend on them directly. With a parent/child split, the logging tasks only see the child's overall outcome, not per-task results.

I'd use this only if you really want one central orchestrator and are fine with that granularity.

3. What I'd do: shrink the wrapper first

Five boilerplate tasks per job is mostly a symptom of putting logic in the graph that belongs in code.

Fold set_params, create_log and the skip decision into one task. It can be a notebook or wheel entry point from a shared library that evaluates the pre-check and emits a task value, with a single condition task branching on it.
Collapse finish_log_success and finish_log_skipped into one finalizer task using run-if dependencies, so it runs on completion regardless of path and records the actual status.

That takes you from 6 wrapper tasks to roughly 3. The logic then lives in one versioned package, not in YAML, so fixing a logging bug doesn't mean regenerating and redeploying every job.

4. Handling deviations without forking

Treat the generator's input as the escape hatch, not the output files.

Never hand-edit resources/*.yml. Anything in there is disposable and regenerated on every build.
Add an explicit overrides section to the entry schema, for example extra_tasks, task_overrides keyed by task_key, cluster_override and skip_wrapper: [finish_log_skipped]. Deviations stay declarative and reviewable, in the same place developers already author config.
Use Jinja macros/blocks in your .j2 templates (one macro per wrapper task, one extension block for custom tasks) rather than one monolithic template per job shape. That also lets you collapse the single/multi templates into one.
Validate before generating. Run a schema check on the entry files, then databricks bundle validate in CI on the generated output. It warns on unknown job properties [2], which catches typos in override blocks.
Define a real exit. If a job needs so many overrides that the template is fighting you, that's a signal to give it a first-class template variant, not a hand-written file. Track how many jobs use each variant.

If you're on a recent CLI, I'd prototype the Python-for-bundles approach with a handful of jobs alongside the Jinja generator. It removes the template-rendering layer and gives you real functions, tests and type checking for the wrapper and the overrides.