Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2022 04:11 PM
Autoscaling doesn't work with structured streaming, so that's not really an option. Autoscaling is based on jobs sitting in the jobs queue for a long time, but that's not the case with streaming. Streaming is more many frequent small jobs.
Spot instances should save money, but you do risk losing VMs if you are outbid. It can also be useful to purchase a lot of VMs ahead of time from the cloud provider. Usually, if you offer to buy many hours you can get a volume discount.
Photon enablement should speed things up and reduce your overall VM need. You'll use more DBUs but on a smaller cluster so you should still save money overall from less cloud costs.