- 1522 Views
- 2 replies
- 0 kudos
KinesisSource generates empty microbatches when there is no new data.
Is it normal for KinesisSource to generate empty microbatches when there is no new data in Kinesis? Batch 1 finished as there were records in kinesis and BatchId 2 started. BatchId 2 was running but then BatchId 3 started . Even though there was no m...
- 1522 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Pranathi Girish​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 0 kudos
- 3361 Views
- 7 replies
- 0 kudos
Resolved! Fetch new data from kinesis for every minute.
I want to fetch new data from kinesis source for every minute. I'm using "minFetchPeriod" option and specified 60s. But this doesn't seem to be working.Streaming query: spark \ .readStream \ .format("kinesis") \ .option("streamName", kinesis_stream_...
- 3361 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Pranathi Girish​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...
- 0 kudos