Here’s how you can implement DQ at each stage:

Bronze Layer

• Checks:
  • File format validation (CSV, JSON, etc.).
  • Schema validation (column names, types).
  • Row count vs. source system.
• Tools:
  • Use Databricks Autoloader with schema evolution and badRecordsPath
  • Implement Great Expectations or Deequ for basic validations.

Silver Layer

• Checks:
  • Remove duplicates.
  • Validate referential integrity (foreign keys).
  • Standardize data types and formats.
• Tools:
  • Delta Live Tables (DLT) with expectations.
  • Great Expectations for advanced profiling.
• Automation:
  • Define expectations in DLT pipelines (expectations block).
  • Fail or quarantine bad records.

Gold Layer

• Checks:
  • Business rule validation (e.g., revenue > 0).
  • KPI consistency checks.
  • Aggregation accuracy.
• Tools:
  • DLT expectations or custom Spark jobs.
  • Integrate with Unity Catalog for governance and lineage.

Practical Tools

• Great Expectations: Flexible, open-source, integrates with Databricks.
• Delta Live Tables: Native expectations for Bronze/Silver/Gold.
• AWS Deequ: For statistical checks.
• Unity Catalog: Governance, lineage, and access control.