Alberto_Umana
Databricks Employee
Databricks Employee

Hello @HoussemBL,

Here are a few points to consider:

  1. Initialization of Generated Columns: Generated columns, such as is_current, rely on the values of other columns (__END_AT in this case) to be correctly populated. During the first run, if the sequencing or initialization of these columns is not handled correctly, it can result in null values.

  2. Sequencing and Ordering: The apply_changes function uses the sequence_by column to determine the order of changes. If the sequencing is not correctly established during the first run, it can lead to issues with the generated columns. Ensure that the sequence_by column (synced in your case) is correctly populated and ordered. 

When you fully refresh the pipeline, it reprocesses the data, which can correct the sequencing and initialization issues, leading to the correct computation of the generated column