Hi @shubham_007 ,

Databricks DLT gives you ability to define data quality rules. You use expectations to define data quality constraints on the contents of a dataset. Expectations allow you to guarantee data arriving in tables meets data quality requirements and provide insights into data quality for each pipeline update. You apply expectations to queries using Python decorators or SQL constraint clauses.

Manage data quality with Delta Live Tables | Databricks on AWS

You can also use open source alternatives. Two best known libraries are:

- Great Expectations

- Soda

Great Expectations

• Python library (retired CLI since April 2023)
• Allows you to define assertions about your data (named expectations)
• Provides a declarative language for describing constraints (Python + JSON)
• Provides expectations gallery with 300+ pre-defined assertions (50+ core)
• A long list of integrations, including data catalogs, data integration tools, data sources (files, in-memory, SQL databases), orchestrators, and notebooks
• Runs data validation using Checkpoints
• Subject matter expert friendly for expectations definition using data assistant
• Automatically generates documentation to display validation results (HTML)
• No official docker image
• Cloud version available
• Great community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)

Soda Core

• CLI tool and Python library
• Allows you to define assertions about your data (named checks)
• Provides a human-readable, domain-specific language for data reliability called Soda Checks Language (YAML)
• Includes 25+ built-in metrics, plus the ability to create user-defined checks (SQL queries)
• Compatible with 20+ data sources (files, in-memory, SQL databases)
• Runs data validation using scans
• Display scan results in the CLI (save to file available) or access them programmatically
• Collects usage statistics (you can opt out)
• Docker image available
• Cloud version available
• Decent community regarding contributions (GitHub), knowledge exchange and Q&A (Slack)