can we commit offset in spark structured streaming in databricks.

[Nis]

We are storing offset details in checkpoint location wanted to know is there a way can we commit offset once we consume the message from kafka.

[raphaelblg · Accepted solution]

Hi @Nis , 

Spark Structured Streaming manages which offsets are consumed internally, rather than rely on the kafka Consumer to do it. Spark manages the source offsets and write them to the spark streaming query checkpoint.

So the answer is no, you cannot commit a Kafka offset through a spark structured streaming query.

Might be worth checking https://stackoverflow.com/questions/50844449/how-to-manually-set-group-id-and-commit-kafka-offsets-i....