Performance issue when using structured streaming

pvaz — Tue, 04 Feb 2025 10:13:36 GMT

Hi databricks community! Let me first apology for the long post.

I'm implementing a system in databricks to read from a kafka stream into the bronze layer of a delta table. The idea is to do some operations on the data that is coming from kafka, mainly filtering and parsing its content to a delta table in unity catalog. To do that I'm using spark structured streaming.

The problem is that I fell that I'm doing something wrong because the number of message that I can process per second seems to low to me. Let me get into the details.

I have a kafka topic that receives a baseline of 300k messages per minute (~ 6MB ) with peaks up to 10M messages per minute. This topic has 8 partitions.

Then I have a job compute cluster with the following configurations:
- Databricks runtime 15.4 LTS
- Worker type Standard_F8 min workers 1 max workers 4
- Driver type Standard_F8

In the cluster I only run a task which takes the data from the kafka cluster, does some filtering operations, including one from_json operation, and stores the data to a unity table. The structured stream is set to be triggered every minute and has the following configurations:

"spark.sql.streaming.noDataMicroBatches.enabled": "false",

"spark.sql.shuffle.partitions": "64",

"spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled": "true",

"spark.sql.streaming.stateStore.providerClass": "com.databricks.sql.streaming.state.RocksDBStateStoreProvider",

"spark.sql.streaming.statefulOperator.stateRebalancing.enabled": "true",

"spark.sql.streaming.stateStore.rocksdb.changelogCheckpointing.enabled": "true"
"maxOffsetsPerTrigger": 10000000
All the other properties are the default values.

I have set a the maxOffsetsPerTrigger in order to prevent out of memory issues in the cluster.

Right now, with those configurations, I can at maximum process about 4M messages per minute. This means that the stream that should run every minute takes more than two minutes to complete. What is strange is that only two nodes of the job compute are active (32GB, 16 cores) with CPU on 10%.

Although this is enough during normal operations I have a lot of unprocessed messages in the back log that I would like to process faster. Does this throughput seems reasonable to you? It feels like I just need to process so little data and even this is not working. Is there anything I can do to improve the performance of this kafka consumer. Thank you for your help.

Re: Performance issue when using structured streaming

NandiniN — Thu, 01 May 2025 06:59:37 GMT

Checking.

Re: Performance issue when using structured streaming

NandiniN — Thu, 01 May 2025 11:46:38 GMT

Have you tried using minPartitions

Minimum number of partitions to read from Kafka. You can configure Spark to use an arbitrary minimum of partitions to read from Kafka using the minPartitions option. Normally Spark has a 1-1 mapping of Kafka topicPartitions to Spark partitions consuming from Kafka. If you set the minPartitions option to a value greater than your Kafka topicPartitions, Spark will divvy up large Kafka partitions to smaller pieces. This option can be set at times of peak loads, data skew, and as your stream is falling behind to increase processing rate. It comes at a cost of initializing Kafka consumers at each trigger, which may impact performance if you use SSL when connecting to Kafka.

topic Re: Performance issue when using structured streaming in Data Engineering

Performance issue when using structured streaming

Re: Performance issue when using structured streaming

Re: Performance issue when using structured streaming