topic Re: data quality check in data engineering in Get Started Discussions

data quality check in data engineering

ankris — Wed, 27 Sep 2023 09:35:36 GMT

Can we use deequ library with azure databricks ? if yes Please provide some support material or examples

Is there any similar data quality library or suggestion to achieve automatic data quality check during data engineering (Azure databricks)

Thanks in advance,

Anantha

Re: data quality check in data engineering

daniel_sahal — Thu, 28 Sep 2023 05:46:41 GMT

@ankris Never used deequ, so cannot say anything about this library but I've used great expectations and it works really good.

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/968100988546031/3944681434720936/8836542754149149/latest.html

Re: data quality check in data engineering

ankris — Fri, 29 Sep 2023 05:43:17 GMT

Thank you for prompt reply. I will go through it. Meantime could anyone provide information on Deequ usage with azure databricks. Also any other library/package (equivalent to deequ) can be used in azure databricks.

Re: data quality check in data engineering

BilalAslamDbrx — Fri, 29 Sep 2023 09:03:34 GMT

@ankris can you describe your data pipeline a bit? If you are writing in Delta Live Tables (my recommendation) then you can express data quality checks "in flight" as the pipeline processes data. You can do post-ETL data quality checks (e.g. at the aggregate level) with Lakehouse Monitoring.

Re: data quality check in data engineering

ankris — Mon, 16 Oct 2023 10:33:25 GMT

We have data pipeline build with delta table and orchestrated using ADF. On top of it we would like to build Data quality assessor using external web interface (already build with streamlit app).Also plan to pass some variables using external interface to evaluate data quality.

Re: data quality check in data engineering

CharlesReily — Tue, 16 Jan 2024 13:09:13 GMT

In Databricks, you can install external libraries by going to the Clusters tab, selecting your cluster, and then adding the Maven coordinates for Deequ.

In your notebook or script, you need to create a Spark session with the Deequ library added as a dependency. This can be done using the spark.jars.packages configuration option.

spark.conf.set("spark.jars.packages", "com.amazon.deequ:deequ:1.4.0")

Write your data quality checks using Deequ functions. For example:

import com.amazon.deequ.{VerificationSuite, VerificationResult}
import com.amazon.deequ.VerificationSuite._

val verificationResult: VerificationResult = VerificationSuite()
.onData(yourDataFrame)
.addCheck(
check = Check(yourColumn, "yourConstraint") // Define your data quality constraint here
)
.run()

Re: data quality check in data engineering

joarobles — Thu, 25 Jul 2024 15:36:09 GMT

Hi there!

You could also take a look at Rudol, it enables no-code Data Quality validations to enable non-technical roles such as Business Analysts or Data Stewards to configure quality checks by themselves.