Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:
-Save money by don't have a cluster up 24/7
-Have bigger files saved for each read
The stream is constant so I cant use once=True because it never ends, that it what I use to read from buckets. The idea is that it reads to the last data available at the moment it started and then gracefully exits.
Can this be done?
Thanks!