Hi,

I was able to solve the issue by specifying maxFetchDuration value more than minFetchPeriod. However I didn't get the purpose of specifying maxFetchDuration value greater than or equal to minFetchPeriod value.

spark \

 .readStream \

 .format("kinesis") \

 .option("streamName", kinesis_stream_name) \

 .option("region", region name) \

 .option("initial Position", "latest") \

 .option("awsAccessKey", awsAccessKey) \

 .option("awsSecretKey", aws_secret_access_key)\

.option("maxFetchDuration","60s")\

.option("minFetchPeriod","60s")