- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2025 08:07 AM
Hi @aliacovella,
That is correct:
-
CDC with Delta Live Tables: The
apply_changesAPI is designed to simplify CDC with Delta Live Tables by processing changes from a change data feed (CDF). This API requires a streaming table to function. -
Streaming Table Requirement: Since streaming tables are not supported from JDBC sources, you cannot directly use a Postgres database as a streaming source for the
apply_changesAPI. -
Alternative Streaming Sources: To implement CDC, you would need to use a streaming source that is supported by Databricks, such as Amazon Kinesis. This would involve setting up a Kinesis stream to capture changes from your Postgres database and then using this stream as the source for your Delta Live Tables pipeline.
-
Configuration and Metrics: When using Kinesis, you can configure various options such as
maxFetchDurationandminFetchPeriodto optimize the streaming query performance. Additionally, Kinesis provides metrics likeavgMsBehindLatest,maxMsBehindLatest, andminMsBehindLatestto monitor the streaming query's progress.