How to run sparkStream for earlier (not future messages)

[alejandrofm]

Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:

-Save money by don't have a cluster up 24/7

-Have bigger files saved for each read

The stream is constant so I cant use once=True because it never ends, that it what I use to read from buckets. The idea is that it reads to the last data available at the moment it started and then gracefully exits.

Can this be done?

Thanks!

[Kaniz]

Hi @Alejandro Martinez​ , Here is a somewhat similar issue on S.O. Please look and let us know if that helped.

[alejandrofm]

Hi, thanks for the link, the solution in that thread is for Kafka and It seems to be a different issue.

I need to stop the process when I reach the events that were available at the time I started listening, so the process will complete NOT when the queue is empty but when I reached a specific point in time on the queue.

Thanks!