Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.
You could also apply data quality checks using open source libraries such as Great Expectations or pydq on the ETL level.
Another approach is to use no code platforms like Rudol to allow non-technical roles such as Data Stewards to implement data quality validations by themselves.
