08-24-2023 09:30 PM
@Agus1
Could you try using CDC in delta. You could use readChangeFeed to read only the changes that got applied on the source table. This is also explained here.
https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
