DLT Pipeline Stats on Object level

[ankit001mittal]

Hi Guys,

I want to create a table where I want to store information about each DLT pipelines on object/table id level details about how much time it took for waiting for resources and how much time it took to run for each object and numbers or records upserted and these kind of information.

From where can I get these details? is there any tables in system catalog where I can get all the information related to DLT pipelines and DLT managed streaming tables and materialised views?

Appreciate your help on this.

[aayrm5]

Hi @ankit001mittal 

DLT Event logs helps you to gather most of the information you've mentioned above. 

Below is the documentation to the DLT Event Logs:

https://docs.databricks.com/aws/en/dlt/observability

Let me know if any questions.

Best,