@ismaelhenzel 

Was working on  similar use case

The use case i was working on :

A source system generates raw Call Detail Records (CDRs) for every call, text, and data session on the network. Your goal is to build a reliable pipeline that cleans and prepares this data for a downstream fraud analytics team.

Solution:

• Define the Data Contract
  Create a YAML file (cdr_contract.yaml) that serves as the source of truth for: Data schema (column names, types, constraints), Data quality rules (null checks, valid ranges, regex patterns, etc.) Store this YAML file in your Git repository alongside your pipeline code for version control and traceability.
• Create the DLT Python Notebook
  In your Databricks workspace, create a Python notebook (process_cdrs.py).This notebook will: Read and parse cdr_contract.yaml. Implement ingestion, transformations, and data quality checks.
• Implement the Bronze and Silver Layers. Bronze Layer: Ingest raw CDRs exactly as received. Preserve original data for replayability and auditing. Silver Layer: Apply transformations and validations based on cdr_contract.yaml. Standardize formats, remove duplicates, and enforce quality rules.
• Configure and Run the DLT Pipeline
  Create a new pipeline and: Point it to your DLT notebook (process_cdrs.py).Configure the target schema/database. Start the pipeline to begin processing data.
• Monitor Data Quality
  In the DLT UI, select your pipeline to view the graph. Review metrics for each quality rule from cdr_contract.yaml: Records passed Records failed. Update the YAML contract as rules evolve and re-run the pipeline to see metrics update automatically.