Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2022 05:04 AM
it depends on the dependencies.
if gold_table1 is dependent on only silver_table1 I'd do
pipeline1 = silver_table1 -> gold_table1 (sequential). Use a cluster pool so you can use warm workers for gold_table1
And in parallel you could do the same for table2. (or even run then all sequentially)
you can also run multiple notebooks in parallel on the same cluster: