Alberto_Umana
Databricks Employee
Databricks Employee

Hi @T0M,

It is expected that the cluster restarts with each validation to ensure that your changes are accurately reflected. 

  • In development mode, once you click "Validate," a cluster will spin up, and it stays active for up to two hours by default. This mode is optimized for quickly detecting and fixing errors by reusing clusters to avoid the overhead associated with frequent restarts.
     
  • However, if you make changes to the pipeline, and then click "Validate" again, the cluster will terminate and a new one will spin up to apply those changes. This is necessary to make sure the pipeline processes the updated code correctly

It is recommended to make larger batch changes and validate them in one go to minimize cluster restarts