Raman_Unifeye
Honored Contributor III

Very Broad Topic. Let me try to break it and provide few key-points.

The most practical design involves defining Data Quality Expectations (rules) in DLT for each layer and implementing an automated process to validate the data against those rules. 

Bronze: Focus on Completeness and Availability

The Bronze layer is your raw, immutable landing zone. The goal is to capture everything and avoid dropping data. Data Quality checks here are minimal and focus on the integrity of the ingestion process itself.

Silver: Focus on Validity, Consistency, and Uniqueness

The Silver layer is where raw data is cleaned, validated, conformed, and enriched. This is the most crucial stage for implementing business-specific quality rules.

Gold: Focus on Accuracy and Business Logic

The Gold layer is for final, aggregated, and curated business-ready data. Checks here confirm that the final transformation and aggregation logic is correct.

Reference Link for DLT/LDP - https://docs.databricks.com/aws/en/ldp/expectations

 


RG #Driving Business Outcomes with Data Intelligence