Anonymous
Not applicable

Yes, it is normal for a KinesisSource to generate empty microbatches when there is no new data in the Kinesis stream. This behavior is a consequence of how the KinesisSource is designed to work.

The KinesisSource continuously polls the Kinesis stream for new data in order to create microbatches to be processed by the Spark Streaming job. If there is no new data available in the stream, the KinesisSource will still create microbatches with no data in them. These empty microbatches are then processed by the Spark Streaming job, which will typically result in no output being generated.

About your question, it is possible that BatchId 3 was triggered because the KinesisSource continued to poll the Kinesis stream after BatchId 2 was created and processed, and it found new data that arrived after BatchId 2 had finished processing. This would result in the KinesisSource creating a new microbatch (BatchId 3) with the new data that had arrived, even though there was no new data available when BatchId 2 was created.