BS_THE_ANALYST
Databricks Partner

@shan-databricks there's certainly ways for the schema to evolve within your delta tables that's supported out of the box: https://docs.databricks.com/aws/en/delta/update-schema#enable-schema-evolution 

To update older records, they'd likely have NULL values in the new column. You could use an UPDATE statement as a one-off to populate them. You could also look into a MERGE statement that is very flexible https://docs.databricks.com/aws/en/sql/language-manual/delta-merge-into#when-matched . This part requires you to write some logic. If you're not confident, you could always use AI + the Databricks Free Edition to practice on in your own environment before implementing it on the real thing.

All the best,
BS