topic Re: Default maximum spark streaming chunk size in delta files in each batch? in Data Engineering

Default maximum spark streaming chunk size in delta files in each batch?

Data_Engineer3 — Sun, 02 Apr 2023 16:20:18 GMT

working with delta files spark structure streaming , what is the maximum default chunk size in each batch?

How do identify this type of spark configuration in databricks?

#[Databricks SQL] #[Spark streaming] #[Spark structured streaming] #Spark

Re: Default maximum spark streaming chunk size in delta files in each batch?

NandiniN — Mon, 03 Apr 2023 14:26:55 GMT

Hello @KARTHICK N ,

The default value for spark.sql.files.maxPartitionBytes is 128 MB. These defaults are in the Apache Spark documentation https://spark.apache.org/docs/latest/sql-performance-tuning.html (unless there might be some overrides).

To check the configurations you can navigate to the Environment tab of the Spark UI and check for the config.

Hope that helps.

Thanks & Regards,

Nandini

Re: Default maximum spark streaming chunk size in delta files in each batch?

Data_Engineer3 — Wed, 05 Apr 2023 05:03:20 GMT

Thanks @Nandini N reply,

I couldn't see this configuration params in databricks job-cluster spark UI, We are using job-cluster for streaming jobs and I don't see this configuration in environment tab in spark UI page.

Is this applicable for streaming concept (because we are using streaming with foreachbatch concept in our project)?

Could you help me to figure it out?

#[Databricks SQL] #[Azure databricks]

Re: Default maximum spark streaming chunk size in delta files in each batch?

Data_Engineer3 — Thu, 29 Aug 2024 15:24:44 GMT

@NandiniN , I couldn't able to see the setting for structured read stream batch size control during the processing the data by using foreachbatch.

Is this possible to control the read stream by records count per each batch in structured streaming?

Re: Default maximum spark streaming chunk size in delta files in each batch?

NandiniN — Thu, 31 Oct 2024 10:00:46 GMT

maxFilesPerTrigger: This option specifies how many new files should be considered in every micro-batch. The default value is 1000.
maxBytesPerTrigger: This option sets a soft maximum on the amount of data processed in each micro-batch. It is not set by default but can be configured to limit the data processed per batch.

Re: Default maximum spark streaming chunk size in delta files in each batch?

NandiniN — Thu, 31 Oct 2024 10:02:59 GMT

doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html

Also, what is the challenge while using foreachbatch?