Re: Best practices for ensuring data quality in ba...

saurabh18cs · ‎01-24-2025

how about adding data profiling or data consistency checks to your batch pipeline?

Few Examples:

# Data profiling

df.describe().show()

# Data consistency checks

# Check for null values

null_counts = df.select([count(when(col(c).isNull(), c)).alias(c) for c in df.columns])

null_counts.show()

# Remove duplicates

df = df.dropDuplicates()

# Handle missing values

df = df.fillna({"age": 0, "email": "unknown@example.com"})

# Standardize data

df = df.withColumn("email", lower(col("email")))

# Data quality metrics

total_records = df.count()

null_email_count = df.filter(col("email").isNull()).count()

print(f"Total records: {total_records}, Null email count: {null_email_count}")