Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Showing results for 
Search instead for 
Did you mean: 

Delta Table Optimize Error

New Contributor III

I have have started getting an error message when running the following optimize command:-



java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Number of records changed after Optimize. NumRecordsCheckInfo(OPTIMIZE,394,1058,2554337689,2600474509,0,0,Map(predicate -> "[]", zOrderBy -> "[]", batchId -> "0", auto -> false))

What is the cause of this? It has been running fine for months.

This is with runtime 11.3 using pyspark.


Esteemed Contributor III
Esteemed Contributor III

Hi, looks like this has to be changed in the command.

Also, could you please recheck if


Also, please tag @Debayan​ with your next response which will notify me. Thank you!

Not applicable

@Dean Lovelace​ :

The error message suggests that the number of records in the Delta table changed after the optimize() command was run. The optimize() command is used to improve the performance of Delta tables by removing small files and compacting larger ones, which can improve query performance and reduce storage costs. However, if there are concurrent write operations happening while the optimize() command is running, it can cause the number of records to change, which can lead to this error.

To resolve this issue, you may want to consider the following steps:

  1. Check for concurrent write operations: Check if there are any other processes or jobs that are writing to the Delta table while the optimize() command is running. If there are, you may need to temporarily stop these operations to avoid conflicts.
  2. Retry the optimize() command: If you're sure there are no concurrent write operations, you can try running the optimize() command again to see if the issue persists. Sometimes, the error message may be due to a transient issue that is resolved when the command is retried.
  3. Use a different optimize() configuration: You can try using a different configuration for the optimize() command, such as increasing the minFileSize or maxFileSize parameters. This may help reduce the likelihood of conflicts with concurrent write operations.
  4. Perform a full compaction: If the issue persists, you can try running a full compaction instead of an optimized compaction. This will merge all the Delta table files into a single file, which can reduce the likelihood of conflicts with concurrent write operations. However, a full compaction can be more resource-intensive and may take longer to complete.

How can I perform a full compaction?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!