Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.
โข Databricks recommends applying data quality rules on the bronze layer before proceeding to the silver and gold layer. โข The recommended approach involves storing data quality rules in a Delta table. โข The rules are categorized by a tag and are used in dataset definitions to determine which restrictions to apply. โข A table named โrulesโ is created to maintain the data quality rules. โข The rules are defined using SQL constraint clauses. โข A function called โget_rules()โ is created to read the rules from the โrulesโ table and return a Python dictionary containing rules matching the provided tag. โข The dictionary of rules is then applied using the '@dlt.expect_all_*()' decorators to enforce data quality constraints. โข The โget_farmers_market_data()โ function is decorated with the '@dlt.expect_all_or_drop()โ decorator, which applies the data quality constraints defined in the โget_rules()โ function to the โraw_farmers_marketโ table.
You could also apply data quality checks using open source libraries such as Great Expectations or pydq on the ETL level.
Another approach is to use no code platforms like Rudol to allow non-technical roles such as Data Stewards to implement data quality validations by themselves.
Looks nice! However I don't see Databricks support in the docs ๐
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
