SparkJun
Databricks Employee
Databricks Employee

Delta Live Tables (DLT): ref: https://docs.databricks.com/en/delta-live-tables/expectations.html

  • Expectations: DLT allows you to define data quality constraints on datasets using expectations. These expectations can be applied to queries using Python decorators or SQL constraint clauses. Actions for invalid records include warning, dropping, or quarantining them.
  • Advanced Validation: You can perform complex data quality checks by defining materialized views using aggregate and join queries.
  • Portability and Reusability: Data quality rules can be maintained separately from pipeline implementations, stored in a Delta table, and applied using tags.