cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Default maximum spark streaming chunk size in delta files in each batch?

Data_Engineer3
Contributor III

working with delta files spark structure streaming , what is the maximum default chunk size in each batch?

How do identify this type of spark configuration in databricks?

#[Databricks SQL]​ #[Spark streaming]​ #[Spark structured streaming]​ #Spark​ 

5 REPLIES 5

NandiniN
Databricks Employee
Databricks Employee

Hello @KARTHICK N​ ,

The default value for spark.sql.files.maxPartitionBytes is 128 MB. These defaults are in the Apache Spark documentation https://spark.apache.org/docs/latest/sql-performance-tuning.html (unless there might be some overrides).

To check the configurations you can navigate to the Environment tab of the Spark UI and check for the config.

Hope that helps.

Thanks & Regards,

Nandini

Thanks @Nandini N​  reply,

I couldn't see this configuration params in databricks job-cluster spark UI, We are using job-cluster for streaming jobs and I don't see this configuration in environment tab in spark UI page.

Is this applicable for streaming concept (because we are using streaming with foreachbatch concept in our project)?

Could you help me to figure it out?

#[Databricks SQL]​ #[Azure databricks]​

@NandiniN , I couldn't able to see the setting for structured read stream batch size control during the  processing the data by using foreachbatch.

Is this possible to control the read stream by records count per each batch in structured streaming?

NandiniN
Databricks Employee
Databricks Employee
  1. maxFilesPerTrigger: This option specifies how many new files should be considered in every micro-batch. The default value is 1000.
  2. maxBytesPerTrigger: This option sets a soft maximum on the amount of data processed in each micro-batch. It is not set by default but can be configured to limit the data processed per batch.

NandiniN
Databricks Employee
Databricks Employee

doc - https://docs.databricks.com/en/structured-streaming/delta-lake.html 

Also, what is the challenge while using foreachbatch?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group