-werners-
Esteemed Contributor III

it depends on the dependencies.

if gold_table1 is dependent on only silver_table1 I'd do

pipeline1 = silver_table1 -> gold_table1 (sequential). Use a cluster pool so you can use warm workers for gold_table1

And in parallel you could do the same for table2. (or even run then all sequentially)

you can also run multiple notebooks in parallel on the same cluster:

https://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-workflows#run-multiple-noteboo...