Hello, thank you for your question.

Since your source is not a continuous streaming source, you’ll need to implement custom checkpointing manually. The best approach is to store the maximum processed row version in a separate checkpoint table. Here’s a simple strategy:

1. Create a Checkpoint Table
   Maintain a table (e.g., checkpoint_table) with a single row containing the last processed row version.

2. Retrieve the Last Processed Row Version
   Query the checkpoint_table before each run to get the last processed row version.

3. Query New Records
   Fetch records where row_version > last_processed_version.

4. Process the Data
   After processing, update checkpoint_table with the new max row_version from the processed data.

You can implement this in a Databricks notebook using Delta Lake tables to ensure ACID compliance. While Delta Live Tables (DLT) has built-in checkpointing for streaming sources, this manual checkpointing approach is necessary for batch ingestion. Let me know if you need more details on implementation!