https://community.databricks.com/t5/data-engineering/how-to-run-sparkstream-for-earlier-not-future-messages/m-p/19620#M13174 <P>Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:</P><P>-Save money by don't have a cluster up 24/7</P><P>-Have bigger files saved for each read</P><P></P><P>The stream is constant so I cant use once=True because it never ends, that it what I use to read from buckets. The idea is that it reads to the last data available at the moment it started and then gracefully exits.</P><P></P><P>Can this be done?</P><P>Thanks!</P><P></P> Wed, 25 May 2022 21:33:17 GMT alejandrofm 2022-05-25T21:33:17Z https://community.databricks.com/t5/data-engineering/how-to-run-sparkstream-for-earlier-not-future-messages/m-p/19620#M13174 <P>Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:</P><P>-Save money by don't have a cluster up 24/7</P><P>-Have bigger files saved for each read</P><P></P><P>The stream is constant so I cant use once=True because it never ends, that it what I use to read from buckets. The idea is that it reads to the last data available at the moment it started and then gracefully exits.</P><P></P><P>Can this be done?</P><P>Thanks!</P><P></P> Wed, 25 May 2022 21:33:17 GMT https://community.databricks.com/t5/data-engineering/how-to-run-sparkstream-for-earlier-not-future-messages/m-p/19620#M13174 alejandrofm 2022-05-25T21:33:17Z https://community.databricks.com/t5/data-engineering/how-to-run-sparkstream-for-earlier-not-future-messages/m-p/19622#M13176 <P>Hi, thanks for the link, the solution in that thread is for Kafka and It seems to be a different issue.</P><P>I need to stop the process when I reach the events that were available at the time I started listening, so the process will complete NOT when the queue is empty but when I reached a specific point in time on the queue.</P><P>Thanks!</P> Thu, 26 May 2022 18:15:00 GMT https://community.databricks.com/t5/data-engineering/how-to-run-sparkstream-for-earlier-not-future-messages/m-p/19622#M13176 alejandrofm 2022-05-26T18:15:00Z