Testing framework using Databricks Notebook and Pytest.

[Anonymous]

Hi Friends,

I am designing a Testing framework using Databricks and pytest. Currently stuck with report generation, that is generating blank with only default parameters only .

for ex :-

testsuites>

<testsuite name="pytest" errors="0" failures="0" skipped="0" tests="0" time="0.157" timestamp="2023-04-11T14:30:47.707323" hostname="XXXXX"/>

</testsuites>

Looking for any other framework which can solve my Data Testing.

[Anonymous]

@Vijaya Palreddy​ :

There are several testing frameworks available for data testing that you can consider using with Databricks and Pytest:

1. Great Expectations: Great Expectations is an open-source framework that provides a simple way to create and maintain automated data tests. It supports a variety of data sources, including Databricks, and provides customizable reports for data quality checks.
2. Apache Griffin: Apache Griffin is another open-source framework that supports data quality testing for various data sources, including Databricks. It provides customizable reports for data quality checks and supports multiple data formats.
3. Data testing framework by Databricks: Databricks also offers a data testing framework that can be used with Pytest. It provides data validation tests for data pipelines, ETL jobs, and data lakes. The framework integrates with Databricks Delta and Apache Spark and generates detailed reports on data quality checks.

You can choose the framework that best fits your needs and integrates well with Databricks and Pytest.