my guess is this because of how the topic is partitioned. Check the docs about availableNow:

Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.

https://spark.apache.org/docs/3.1.3/structured-streaming-kafka-integration.html