So furthermore, my temp solution as you may have noticed includes all version of the Delta tables & contains N number of duplicate records.

This is what I have in my production systems now.

sql = f"CREATE TABLE IF NOT EXISTS {deltalake_glue_database_name}.{table_name} USING DELTA LOCATION '{location}'"
spark.sql(sql)

It runs every batch. I'm still conducting more validation as I'm still getting reported numbers being a little off.