cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pranathisg97
by New Contributor III
  • 977 Views
  • 2 replies
  • 0 kudos

KinesisSource generates empty microbatches when there is no new data.

Is it normal for KinesisSource to generate empty microbatches when there is no new data in Kinesis? Batch 1 finished as there were records in kinesis and BatchId 2 started. BatchId 2 was running but then BatchId 3 started . Even though there was no m...

  • 977 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Pranathi Girish​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
1 More Replies
alejandrofm
by Valued Contributor
  • 631 Views
  • 2 replies
  • 0 kudos

How to run sparkStream for earlier (not future messages)

Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:-Save money by don't have a cluster up 24/7-Have bigger files saved for each readThe stream is constant so...

  • 631 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Alejandro Martinez​ , Here is a somewhat similar issue on S.O. Please look and let us know if that helped.

  • 0 kudos
1 More Replies
Labels