bianca_unifeye
Databricks MVP

Pipeline cost

  • ADF charges per activity run + Data Integration Units (DIUs) used for copy/mapping activities, plus Integration Runtime time.

  • LakeFlow charges per compute usage, not per-step. Fewer moving pricing knobs, but your cost is dominated by:

    • cluster size/type

    • runtime (how long the job runs)

    • how often you trigger it

You will pay also for storage.

For many migrations, you replace a large ADF pipeline full of small activities with one or a few LakeFlow jobs, so you trade lots of per-activity charges for a simpler, cluster-based cost model.

Job reuse 

Within a single LakeFlow job run you can reuse the same compute across multiple notebook tasks – if you configure it that way. Each job run itself is independent.

  • A LakeFlow job is a DAG of tasks (notebooks, pipelines, Python scripts, etc.).

  • Each task has a compute configuration:

    • shared job cluster (recommended), or

    • its own cluster, or

    • serverless, or

    • an existing interactive cluster.

If you attach all notebook activities in that job to the same job cluster / serverless definition, then in one run:

  • the cluster is started once,

  • all tasks run on that same cluster,

  • the cluster is then terminated based on your settings.

 

It does not automatically reuse compute across different job runs or across different jobs, unless you deliberately target a long-running interactive cluster.

For an ADF pipeline with N Databricks notebook activities, the usual pattern is:

  • create one LakeFlow job with N notebook tasks,

  • attach them all to one shared job cluster → this typically reduces cost vs many independent jobs.