DLT pipeline observability questions (and maybe suggestions)

guangyi
Contributor III

All my questions is around this code block

@Dlt.append_flow(target=”target_table”):
def flow_01():
  df = spark.readStream.table(“table_01”)

@dlt.append_flow(target=”target_table”):
def flow_02():
  df = spark.readStream.table(“table_02”)

The first question is can I manually check or update or delete the checkpoint of stream table reading in the above code?

I suppose there is no way of specifing the checkpoint location because I cannot find any document about this feature. I try to find resource under the Structured Streaming checkpoints or Configure a Delta Live Tables pipeline but nothing there.

Why I want to do this is because I want to do some troubleshooting. For example, I want to monitor the daily data reading from a specify streaming table, like how much data has been read this time, start from where ended at where. Also if something wrong goes wrong, I can delete the checkpoint as a reset measurement. There is a full refresh can solve this problem but if I can access to the checkpoint it may give me lots insight about the pipeline is running.

The second question is similar, is there a way to monitor the append_flow behavior, like how much data flow the source table to the target table on the daily job?

The reason I want this is because, after all the append flow process accomplished what I got from the target table is only a total incremental number for this time. I cannot differentiate each individual flow contributing how much data or how much time they cost individually

Is there any feature I ignored or I can referenced to accomplish my goal? Or do we have any plan to achieve these feature in the future?