Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2025 07:48 AM
We have 3 possible compute clusters that we can run a notebook against.
They are varying sizes and the one that the notebook uses will depend on the size of the data being processed.
We "t-shirt size" each tenant base on their data size (S, M, L) and can read this config in from Postgres in a notebook.
Once we know the t-shirt size, is there a way of setting the compute cluster dynamically in subsequent tasks?
e.g. a tenant is size M so the rest of the tasks in the job run on the M cluster
We'd like to avoid duplicating jobs/tasks!
Thanks in advance