https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20254#M13646 <P>Looking at the SQL Job and watching 309 mil rows and 55 hr of run time while stream status is still initializing. No data has been written to a table which is the end of the process as well.</P> Tue, 29 Nov 2022 22:25:49 GMT AdamRink 2022-11-29T22:25:49Z https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20252#M13644 <P>I have a large stream of data read from Confluent Kafka, 500+ millions of row. When I initialize the stream I cannot control the batch sizes that are read.</P><P></P><P>I've tried setting options on the readstream - maxBytesPerTrigger, maxOffsetsPerTrigger, fetch.max.bytes, max.poll.records</P><P></P><P>Configuring spark cluster options maxRatePerPartition</P><P></P><P>Starting with a fresh checkpoint</P> Mon, 28 Nov 2022 14:03:26 GMT https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20252#M13644 AdamRink 2022-11-28T14:03:26Z https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20253#M13645 <P>Hi @Adam Rink​&nbsp;</P><P>Just checking for further info on your question. How are you deducing that the batch sizes are more than what you are providing as maxOffsetsPerTrigger ?</P> Tue, 29 Nov 2022 18:46:52 GMT https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20253#M13645 UmaMahesh1 2022-11-29T18:46:52Z https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20254#M13646 <P>Looking at the SQL Job and watching 309 mil rows and 55 hr of run time while stream status is still initializing. No data has been written to a table which is the end of the process as well.</P> Tue, 29 Nov 2022 22:25:49 GMT https://community.databricks.com/t5/data-engineering/how-to-limit-batch-size-from-confluent-kafka/m-p/20254#M13646 AdamRink 2022-11-29T22:25:49Z