Hi All,
We are working on creating a DQ framework on DLT pipelines in Databricks.
Databricks DLT pipelines reads incoming data from Kafka / Files sources. once data is ingested Data validation must happen on top of the ingested data. Customer is evaluating if Expectations or Great Expectations will work on streaming real time pipelines. The DQ rules will be applied on top of the incoming data as a chunk read per event and not on individual rows.
