Hi Kash, on 4th point, do you guys have realtime ingestion to model ? or its batch. in case of batch, DLT will be fine i guess. but would love to know more. never seen realtime model updates ealier.
There are many DQ tools and platforms, but most are SQL based, and thus it costs and its delayed. so it really depends on your use-case and problem statement. sometimes it makes sense to build your own, but most of the time it does not make sense if...
all those DQ tools work on SQL architecture, so its not built for streaming, also its not built for batch dataset with efficiency and complex DQ checks in mind. this is why we built most comprehensive data quality/monitoring platform, happy to share ...
GE and other DQ tools will fire lot of SQLs, increasing cost and adding delays. so it depends on whats your requirements are. happy to discuss more if you are interested, as I am also going to make such tool available to databricks community as well ...
