Hi, I've dug out a couple of articles, that I think may be a good starting point for you but I think one of the key things to think about is how to store and manage your rules. Most teams start with Yaml based configuration and use their source control to manage it. This works really well if it will be data engineers managing the rules. If however you want data owners and custodians to define their own rules then you may want to look at storing them in UC tables, you could then build some kind of app for people to be able to add rules. I think the key with a data quality framework is to start small and learn lessons along the way. Perhaps target your 10 most high profile tables to start off with and go from there.

Articles:

https://www.advancinganalytics.co.uk/blog/databricks-data-quality-framework-dqx-insurance-use-case

https://medium.com/@vsanmed/preventing-data-disasters-a-guide-to-proactive-quality-checks-with-dqx-o...

I hope this helps.

Many Thanks,

Emma