How to run sparkStream for earlier (not future messages)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-25-2022 02:33 PM
Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:
-Save money by don't have a cluster up 24/7
-Have bigger files saved for each read
The stream is constant so I cant use once=True because it never ends, that it what I use to read from buckets. The idea is that it reads to the last data available at the moment it started and then gracefully exits.
Can this be done?
Thanks!
- Labels:
-
Kinesis and spark streaming
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-26-2022 11:15 AM
Hi, thanks for the link, the solution in that thread is for Kafka and It seems to be a different issue.
I need to stop the process when I reach the events that were available at the time I started listening, so the process will complete NOT when the queue is empty but when I reached a specific point in time on the queue.
Thanks!
