topic How to get the count of dataframe rows when reading through spark.readstream using batch jobs? in Data Engineering

How to get the count of dataframe rows when reading through spark.readstream using batch jobs?

SRK — Wed, 21 Dec 2022 16:29:40 GMT

I am trying to read messages from kafka topic using spark.readstream, I am using the following code to read it.

My CODE:

df = spark.readStream

.format("kafka")

.option("kafka.bootstrap.servers", "192.1xx.1.1xx:9xx")

.option("subscribe", "json_topic")

.option("startingOffsets", "earliest") // From starting

.load()

Now i just want to get the count of df like we can get from df.count() method when we use spark.read.

I need to place some conditions if i didn't get any messages from the Topic. I am running this code as a batch and its a business requirement, i don't want to use spark.read.

Please suggest what would be the best approach to get the count.

Thanks in advance!

Re: How to get the count of dataframe rows when reading through spark.readstream using batch jobs?

daniel_sahal — Thu, 22 Dec 2022 13:13:54 GMT

You can try this approach:

https://stackoverflow.com/questions/57568038/how-to-see-the-dataframe-in-the-console-equivalent-of-show-for-structured-st/62161733#62161733

ReadStream is running a thread in background so there's no easy way like df.show().

Re: How to get the count of dataframe rows when reading through spark.readstream using batch jobs?

SRK — Fri, 23 Dec 2022 06:07:16 GMT

Thanks for the suggestion. I will check.