Many of our source data are setup in a way that the main table only keep small amount of data, and historical data are move to another archive table with very similar schema.
My goal is have one table in Databricks, maybe with a flag to indicate if the data is current or archived.
How should I setup my pipeline to support this, and is there a best practice that I should follow?
