Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2024 01:50 AM
Hello @guangyi , I am getting back to you with some insights
- Regarding your first question about checkpointing
- You can manually check the checkpointing location of your stream table. The checkpoints of your Delta Live Tables are under Storage location in Destination of Pipeline settings. Each table gets a dedicated directory under <storage_location>/checkpoints/<dlt_table_name>
- As for update or delete the checkpointing, it's not technically feasible AFAIK, and it's not something that we would recommend either. It could create some unexpected behavior of your stream pipelines, and it'll be difficult to troubleshoot this type of error.
- To closely monitor your streaming pipelines, there are some useful information that you can find in the event log of Delta Live Tables (some example here https://docs.databricks.com/en/delta-live-tables/observability.html). For example, you can query the event log (it can either be in Hive metastore or Unity Catalog) and find the timestamp where each events is processed. In addition, you can create your custom monitoring rules via event hooks https://docs.databricks.com/en/delta-live-tables/event-hooks.html
- For your second question on the append flow, there isn't any out-of-the-box solution to differentiate the contribution of different flows, but I encourage you to look into the event log of the target table. You can check the event log of your table in Unity Catalog by querying
SELECT * FROM event_log(TABLE(my_catalog.my_schema.table1))
I hope that my answers are clear and can address some of your points. If you want to discuss further, please kindly let me know!