https://community.databricks.com/t5/data-engineering/tracking-dbms-cdc/m-p/62595#M32007 <P>We're using Databricks to incrementally extract data from SQL Server tables into S3. The data contains a timestamp column. We need a place to store the maximum retrieved timestamp per table so it can retrieved during the next run.</P><P>Does Databricks contain (or easily connect to) any key-value stores, or have similar functionality? It could of course be tracked using a Delta Lake table but implementing a "frequent updates by primary key" pattern in a columnar storage system seems like a bad idea.</P> Mon, 04 Mar 2024 14:49:08 GMT demost11 2024-03-04T14:49:08Z https://community.databricks.com/t5/data-engineering/tracking-dbms-cdc/m-p/62595#M32007 <P>We're using Databricks to incrementally extract data from SQL Server tables into S3. The data contains a timestamp column. We need a place to store the maximum retrieved timestamp per table so it can retrieved during the next run.</P><P>Does Databricks contain (or easily connect to) any key-value stores, or have similar functionality? It could of course be tracked using a Delta Lake table but implementing a "frequent updates by primary key" pattern in a columnar storage system seems like a bad idea.</P> Mon, 04 Mar 2024 14:49:08 GMT https://community.databricks.com/t5/data-engineering/tracking-dbms-cdc/m-p/62595#M32007 demost11 2024-03-04T14:49:08Z