Therefore, I insert the pipeline diagnostics into the Storage --> A Databricks notebook is automatically triggered --> I can process the data in this notebook using filters and then insert it into the Delta Table.

Is it something like this?