Thank you for the extra perspectives!

Yes, there is already a volume discounts (VM resources purchased ahead of time) negotiated by the company and the cloud provider. Sorry I left that out as I was too focused on the technical options.

I am on the same page with you that both autoscaling and spot instances are not compatible with my structured streaming workloads and SLAs. For example, in the scenario of recovering from a "spot instance" being outbid, I have to account for the X amount of time for the next nodes to be available and Y amount of time for those nodes to complete the bootstrap sequence (getting imaged and added to my cluster). The variable time (sum of X + Y) would already have pushed my streaming workload out of SLA and source data backlog to build up.

Learning from you, the only viable variable that I have not explored so far is Photon.

Is there a general rule of thumb to understand how Photon can size down from non-Photon cluster/workload? For example, Photon reduce physical memory requirements by 20% from non-Photon workloads if the workloads are identical. But the CPU cores should remain the same. <-- Of course, this is a complete fabrication here, but the idea is I am looking for such mapping and translation so I know how to optimize my cluster VM sizing for Photon runtime. Any references are appreciated. <-- Happy to make this into a new question if the amount of effort is going to go beyond the scope of this original question. For example: how do I optimize my compute (streaming) workload when using Photon runtime. Please let me know. Thanks!