https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/ba-p/141683 <H1><SPAN>Introduction</SPAN></H1> <P><A href="https://www.databricks.com/product/data-engineering" target="_blank" rel="noopener"><SPAN>Databricks Lakeflow</SPAN></A><SPAN> enables data teams to design and operate data pipelines at scale, where speed and reliability directly influence the time to market for insights. As pipeline complexity grows, test automation becomes essential to maintaining data quality and ensuring smooth, predictable production workflows. With </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/" target="_blank" rel="noopener"><SPAN>Databricks Asset Bundles</SPAN></A><SPAN>, setting up CI/CD and automated testing on Databricks has become significantly simpler, empowering teams to build with confidence and deliver value faster.</SPAN></P> <P><SPAN>Different layers of testing play a role in building reliable data pipelines:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Unit tests validate individual functions or transformations and can run locally (in an IDE or CI agent) using Apache Spark™ and other dependencies in local mode, which is ideal for providing fast feedback during development. While unit tests are lightweight and efficient, they cannot cover certain Databricks-specific capabilities, especially newer features not yet available in open source Spark, Delta, or Unity Catalog.&nbsp; Additionally, they are not intended for testing interactions with external dependencies like Kafka, REST APIs or databases.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Component tests verify how multiple functions or transformations collaborate within a single bounded data pipeline component, ensuring that local logic behaves as intended before integrating with other parts of the data pipeline.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Integration tests then validate complete workflows running on Databricks, confirming that data dependencies, configurations, and platform features behave correctly end-to-end, making them essential for achieving full reliability.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Acceptance and performance tests validate business requirements and scalability in production-like environments.</SPAN></LI> </UL> <P>In this blog, we focus on integration testing, examining two different approaches, and present a blueprint for one of them.</P> <P>&nbsp;</P> <H1><SPAN>Integration Testing Approaches</SPAN></H1> <P><SPAN>Choosing the right approach for integration testing your Lakeflow jobs in Databricks environments is critical for CI/CD maturity, developer productivity, and increasing confidence when deploying changes to a production environment. Two commonly adopted strategies are: Databricks workflow-based integration testing and local integration tests with tools such as </SPAN><A href="https://pytest.org/" target="_blank" rel="noopener"><SPAN>Pytest</SPAN></A><SPAN>. Let’s dive deep into both approaches to see how they work, and their advantages and disadvantages.</SPAN></P> <H2><SPAN>Approach 1: Using Databricks Lakeflow Jobs for Integration Testing</SPAN></H2> <P><SPAN>In this approach, tests are implemented as Databricks notebooks or Python scripts, orchestrated and scheduled via Databricks Lakeflow Jobs. Tests run within the Databricks cloud environment, accessing clusters, data sources, and job code directly. This method requires developers to create and deploy two jobs for each pipeline: a main ETL job and a dedicated integration testing job. The main job handles the ETL tasks, while the integration testing job orchestrates test notebooks to set up environments, execute main jobs, validate results, and clean up resources.</SPAN></P> <P><SPAN>The approach consists of the following phases, each executed as a separate task within the integration test job:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Setup task: Create isolated test resources (catalogs, schemas, tables) and populate them with test data</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Trigger task: Run the actual Lakeflow job against the prepared test environment</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Validation task: Assert expected outcomes and validate data quality</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Teardown task: Clean up test resources to prevent catalog/workspace clutter</SPAN></LI> </UL> <P><SPAN>Typically, the integration testing job is deployed and run only in specific environments (e.g. test, acceptance) through a CI/CD process. To ensure isolation and environment-specific functionality, the main ETL job must be parameterized, allowing all environment-dependent values (catalogs, schemas, paths) to be configurable via notebook parameters. The CI agent is responsible for deploying, executing, and validating the integration test results, thereby determining if the build is successful or a failure.</SPAN></P> <P><SPAN>However, this approach presents several drawbacks, primarily due to the necessity of creating and deploying a secondary job for testing alongside the main job. These include:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Defining setup and teardown logic</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Defining the secondary integration testing job with the correct tasks and order</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Deploying both jobs to the appropriate environment (either locally or through a CI agent)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Polling and parsing job results</SPAN></LI> </UL> <P><SPAN>Additionally, the approach slows down the feedback loop during both test and job development. Running tests requires deploying both jobs with the right parameters to the right environment, running them, waiting for results, and interpreting those results. This context switching breaks the developer flow.</SPAN></P> <H2><SPAN>Approach 2: Using Pytest for Integration Testing</SPAN></H2> <P><SPAN>In this approach, tests are written locally using Pytest and developed in your preferred IDE with full debugging capabilities. Tests leverage </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/databricks-connect/python/" target="_blank" rel="noopener"><SPAN>Databricks Connect</SPAN></A><SPAN> to establish a remote execution context on Databricks clusters while orchestrating everything from your local machine or a CI agent. Integration tests are written using Pytest, leveraging its capability for setting up and tearing down resources through fixtures, triggering the job to be tested, and validating the results afterward through Pytest assertions.</SPAN></P> <P><SPAN>The typical workflow is:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Test Setup: Pytest fixtures use Databricks Connect to create test-specific resources (catalogs, schemas, tables) using the remote cluster, establishing an isolated test environment.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Job Execution: The Lakeflow job is triggered to run on Databricks, operating against the test data and environment.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Result Validation: After job completion, Pytest assertions query results through Databricks Connect, validating outputs, data quality, and business logic.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Teardown: Test fixtures clean up resources, removing test-created catalogs, schemas, and tables to maintain workspace hygiene and prevent test interference.</SPAN></LI> </UL> <P><SPAN>Similar to the first approach, the main job under test must be parameterized for all environments with test-specific values configured during test execution. This ensures test isolation, as resources are created specifically for each test run. Developers can execute the test either locally or via a CI agent without deploying additional jobs to Databricks, enabling them to assert job execution results remotely. Pytest also offers test reporting, which helps in validating and quickly identifying successful and failing tests.</SPAN></P> <P><SPAN>This approach provides the benefits of:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Rapid Development and Iteration: Tests are developed, executed, and debugged locally without additional deployments or workflow scheduling.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Tight Feedback Loops: Developers see results in the same IDE without the need to switch between different tools and UI during active development.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Complete Pytest Ecosystem Access: Access to fixtures for complex setup/teardown logic, parametrization for testing multiple scenarios, powerful assertions, and hundreds of plugins for specialized testing needs.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Remote Execution: Still executes within the Databricks environment through remote execution using Databricks Connect.</SPAN></LI> </UL> <H1>&nbsp;</H1> <H1><SPAN>Approach 2 Deep Dive</SPAN></H1> <H2><SPAN>Blueprint Overview</SPAN></H2> <P><SPAN>At its core, this blueprint defines a pytest-driven framework to run integration tests on each Lakeflow job (with multiple tasks). Every test case follows a standardized structure focused on reproducibility, isolation, and full verification of pipeline outputs.</SPAN></P> <H2><SPAN>Pytest Template Structure</SPAN></H2> <H3><SPAN>Setup Phase</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Establishes a predefined, isolated clean state by creating new ephemeral Unity Catalog schemas and volumes required for each test run, ensuring every run is consistent and independent of prior (and parallel) executions. Note that Unity Catalog schema and volume creation privileges are required to execute integration tests.</SPAN></LI> </UL> <H3><SPAN>Trigger Phase</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Invokes the deployed Databricks workflow via the Workspace client, initiated from within a pytest fixture.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Supports parameterization to test multiple pipeline configurations, allowing each test to run in parallel and isolation on its own Unity Catalog schema.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The trigger emulates a production job, validating orchestration logic and Databricks-specific dependencies that local Spark tests cannot achieve.</SPAN></LI> </UL> <H3><SPAN>Assertion and Cleanup Phase</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>After workflow completion, assertions validate the resulting state in Unity Catalog tables.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Typical validations include:</SPAN></LI> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Record counts in Delta or Iceberg tables.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Comparing records to stored snapshots.</SPAN></LI> </UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Cleanup logic tears down or deletes the ephemeral resources, such as tables, checkpoints, and volumes.</SPAN></LI> </UL> <P><SPAN>Each test execution provides consistent validation against real Databricks runtime behavior, from workflow orchestration to data persistence, while preserving the speed and productivity of local workflows.</SPAN></P> <H2><SPAN>Orchestrating Tests with databricks-labs-pytester</SPAN></H2> <P><SPAN>The </SPAN><A href="https://github.com/databrickslabs/pytester" target="_blank" rel="noopener"><SPAN>databricks-labs-pytester</SPAN></A><SPAN> package is a valuable utility for orchestrating integration tests with Databricks, enhancing pytest by providing native Databricks capabilities for resource management, Spark session handling and test isolation.</SPAN></P> <H3><SPAN>Key benefits</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Prebuilt fixtures handle environment setup, such as the creation of ephemeral Unity Catalog schemas and volumes required for each test.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Workspace integration exposes convenient APIs (through the </SPAN><SPAN>ws</SPAN><SPAN> fixture) to trigger and monitor Databricks workflows.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Lifecycle consistency guarantees clean setup and teardown execution even when tests fail.</SPAN></LI> </UL> <P><SPAN>Here’s an example integration test showing these concepts in action:</SPAN></P> <LI-CODE lang="python">@pytest.mark.integration_test def test_job(spark, make_schema, ws, job_id): catalog_name = "main" # Ephemeral schema creation schema_name = make_schema(catalog_name=catalog_name).name # Trigger a Databricks Workflow run run_wait = ws.jobs.run_now( job_id=job_id, python_params=[catalog_name, schema_name] ) # Wait for run completion and validate success run_result = run_wait.result() result_status = run_result.state.result_state assert result_status == RunResultState.SUCCESS # Validate result data written to Unity Catalog table df = spark.read.table(f"{catalog_name}.{schema_name}.my_table") assert df.count() &gt; 0 @pytest.fixture def job_id(ws, request): job_name = request.config.getoption("--job-name") job_id = next((job.job_id for job in ws.jobs.list() if job.settings.name == job_name), None) if job_id is None: raise ValueError(f"Job '{job_name}' not found.") return job_id</LI-CODE> <P><SPAN>In this test:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>make_schema</SPAN> <SPAN>creates an ephemeral Unity Catalog schema required for this specific test.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>ws</SPAN> <SPAN>gives a handle to the Databricks Workspace client, allowing direct workflow triggering via </SPAN><SPAN>run_now</SPAN><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Post-execution, data is validated through the </SPAN><SPAN>spark</SPAN> <SPAN>fixture powered by Databricks Connect, demonstrating full end-to-end validation from orchestration to data verification.</SPAN></LI> </UL> <P><SPAN>This minimal example demonstrates both Pytester’s fixture-driven simplicity and its tight integration with Lakeflow jobs, enabling well-readable, fully automated tests optimized for CI/CD.</SPAN></P> <H2><SPAN>Leveraging DB Connect</SPAN></H2> <P><SPAN>While Pytester manages Databricks resource orchestration and improves the test code structure, Databricks Connect powers the actual Spark execution on any Databricks compute, including serverless compute. Databricks Serverless provides instant, on-demand compute, which is critical for fast test completion within both inner and outer development loops.</SPAN></P> <P><SPAN>Through Databricks Connect, tests:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Operate natively with Unity Catalog assets on Databricks.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Execute Spark transformations and queries for setup or assertion phases remotely on the Databricks runtime.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Maintain parity with production configurations without requiring local Spark clusters.</SPAN></LI> </UL> <P>&nbsp;</P> <H2><SPAN>Managing Python Environments with the </SPAN><A href="https://docs.astral.sh/uv/" target="_blank" rel="noopener"><SPAN>uv</SPAN></A><SPAN> package manager</SPAN></H2> <P><SPAN>Unit and integration tests depend on different Spark and Databricks runtime contexts. The UV package manager makes it easy to enforce dependency isolation across these layers through</SPAN> <SPAN>pyproject.toml</SPAN> <SPAN>dependency groups.</SPAN></P> <P><SPAN>Example configuration from</SPAN><SPAN> pyproject.toml</SPAN><SPAN>:</SPAN></P> <LI-CODE lang="markup">[dependency-groups] dev = [ "pytest&gt;=8.3.4", "databricks-labs-pytester" ] unit-tests = [ # These dependencies will break integration tests relying on databricks-connect "pyspark&gt;=4.0.0,&lt;5.0.0" ] integration-tests = [ "databricks-connect==17.1.0" ]</LI-CODE> <H3><SPAN>Usage patterns</SPAN></H3> <P><SPAN>Run integration tests:</SPAN></P> <LI-CODE lang="markup">#bash uv sync --only-group integration-tests uv run python -m pytest -rsx -m integration_test --job-name="&lt;job-name-to-test&gt;"</LI-CODE> <P><SPAN>Run unit tests:</SPAN></P> <LI-CODE lang="markup">#bash uv sync --only-group unit-tests uv run python -m pytest -m unit_test</LI-CODE> <P><SPAN>With isolated dependency groups, data teams can confidently test across multiple layers while maintaining a single, consistent repository. Shared dependencies under the</SPAN> <SPAN>dev</SPAN> <SPAN>group reduce duplication and ensure identical setups both locally and within CI/CD executions.</SPAN></P> <H1>&nbsp;</H1> <H1><SPAN>Demo</SPAN></H1> <P><SPAN>The following section showcases how this setup will run in an actual Databricks environment.&nbsp;</SPAN><SPAN>The full demo code is available in the </SPAN><A href="https://github.com/databricks-solutions/databricks-blogposts/tree/main/2025-12-integration-tests-blueprint-demo" target="_blank" rel="noopener"><SPAN>Databricks Blogposts</SPAN></A><SPAN> GitHub repository.</SPAN></P> <H2><SPAN>Setting up the environment</SPAN></H2> <P><SPAN>Prerequisite:&nbsp; Install and setup the following dependencies</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/dev-tools/cli/install" target="_blank" rel="noopener"><SPAN>Install and set up Databricks CLI locally</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.astral.sh/uv/guides/install-python/" target="_blank" rel="noopener"><SPAN>Install Python uv</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/dev-tools/cli/authentication" target="_blank" rel="noopener"><SPAN>Authenticate the Databricks CLI to your dev workspace</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Install required Python dependencies using </SPAN><SPAN>uv</SPAN></LI> </UL> <LI-CODE lang="markup">#bash uv sync --only-group integration-tests</LI-CODE> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Set up the following environment variables for Pytester</SPAN></LI> </UL> <LI-CODE lang="markup">export DATABRICKS_HOST=&lt;your-dev-workspace-url&gt; export DATABRICKS_CLUSTER_ID=&lt;cluster-id&gt; # Used to run Spark code on Databricks export DATABRICKS_WAREHOUSE_ID=&lt;warehouse-id&gt; # Used by pytester fixtures to run SQL queries # Optionally if you want to use serverless instead of classic compute, replace DATABRICKS_CLUSTER_ID with this export DATABRICKS_SERVERLESS_COMPUTE_ID=auto</LI-CODE> <H2><SPAN>Bundling and deploying the job</SPAN></H2> <P><SPAN>Set up a Databricks Assets Bundle for the main job which will make it easy to define and deploy the job to different target environments.</SPAN></P> <P><SPAN>Example job definition:&nbsp;</SPAN></P> <LI-CODE lang="markup"># ./resources/workflow_test_automation_blueprint.job.yml resources: jobs: workflow_test_automation_blueprint_job: name: workflow_test_automation_blueprint_job tasks: - task_key: calculate_avg_trip_distance python_wheel_task: entry_point: main package_name: ps_test_blueprint parameters: ["main","default"] environment_key: default environments: - environment_key: default spec: environment_version: '3' dependencies: - ../dist/*.whl</LI-CODE> <P><SPAN>Example </SPAN><SPAN>databricks.yml</SPAN><SPAN> file</SPAN></P> <LI-CODE lang="markup">bundle: name: workflow_test_automation_blueprint include: - resources/*.yml artifacts: default: type: whl build: uv build --wheel --package ps_test_blueprint path: . targets: dev: mode: development default: true</LI-CODE> <P><SPAN>Once the YAML files are defined and configured properly, deploy the main job to be tested toward a Databricks environment where you want to test the job. In this example, it will be deployed to a dev environment.</SPAN></P> <LI-CODE lang="markup">databricks bundle deploy -t dev</LI-CODE> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Taras_Chaikovsk_0-1765466452035.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22221iB040759EF4FC0B09/image-size/medium?v=v2&amp;px=400" role="button" title="Taras_Chaikovsk_0-1765466452035.png" alt="Taras_Chaikovsk_0-1765466452035.png" /></span></P> <P>&nbsp;</P> <H2><SPAN>Running the integration tests</SPAN></H2> <P><SPAN>Next, execute the integration tests, which will set up the environment through fixtures, trigger the main job that was deployed in the previous step, and validate the results of the job run. This can be run multiple times for the same job deployed in the previous step.</SPAN></P> <LI-CODE lang="markup">uv run python -m pytest -rsx -m integration_test --job-name="workflow_test_automation_blueprint_job"</LI-CODE> <P><SPAN>Once the test runs, a new test-specific schema is created to store the output tables and is passed to the job as a parameter. The main job started running as well.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Taras_Chaikovsk_1-1765466452036.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22222i921C14EA060B9D1F/image-size/medium?v=v2&amp;px=400" role="button" title="Taras_Chaikovsk_1-1765466452036.png" alt="Taras_Chaikovsk_1-1765466452036.png" /></span></P> <P><SPAN>The schema </SPAN><SPAN>dummy_*</SPAN><SPAN> is created by Pytester fixture specifically for the tests.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Taras_Chaikovsk_2-1765466452037.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22220i1E855DD3106CF84F/image-size/medium?v=v2&amp;px=400" role="button" title="Taras_Chaikovsk_2-1765466452037.png" alt="Taras_Chaikovsk_2-1765466452037.png" /></span></P> <P>&nbsp;</P> <H2><SPAN>Results and cleanup</SPAN></H2> <P><SPAN>The main job ran successfully and the results are returned to the test to be validated. Marking the tests a success and producing the pytest report</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Taras_Chaikovsk_3-1765466452037.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22223i6D01D11647F62290/image-size/medium?v=v2&amp;px=400" role="button" title="Taras_Chaikovsk_3-1765466452037.png" alt="Taras_Chaikovsk_3-1765466452037.png" /></span></P> <P><SPAN>The schema and any resources created specifically for the test are automatically deleted after the test run.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Taras_Chaikovsk_4-1765466452038.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22224i5D8C75AFF788AA29/image-size/medium?v=v2&amp;px=400" role="button" title="Taras_Chaikovsk_4-1765466452038.png" alt="Taras_Chaikovsk_4-1765466452038.png" /></span></P> <P>&nbsp;</P> <H1><SPAN>Conclusion</SPAN></H1> <P><SPAN>In this blog, we explored two approaches for integration testing of Lakeflow Jobs and presented a practical blueprint for Approach 2 using Pytest and Databricks Connect. By combining Pytest’s&nbsp; framework, including advanced fixture management through databricks-pytester, with Databricks Connect’s remote job execution, data teams gain faster feedback cycles, higher developer productivity, and more reliable data pipelines. This streamlined workflow empowers teams to test the entire lifecycle, from orchestration and external dependencies to data persistence, directly within their IDE or CI environment.</SPAN></P> Wed, 17 Dec 2025 14:02:31 GMT Taras_Chaikovsk 2025-12-17T14:02:31Z https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/ba-p/141683 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-12-11 at 16.18.44.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22219iE979563CD7870565/image-size/medium?v=v2&amp;px=400" role="button" title="Screenshot 2025-12-11 at 16.18.44.png" alt="Screenshot 2025-12-11 at 16.18.44.png" /></span></P> Wed, 17 Dec 2025 14:02:31 GMT https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/ba-p/141683 Taras_Chaikovsk 2025-12-17T14:02:31Z https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/149712#M944 <P>Thanks for this very helpful and informative blog post.</P><P>I’d appreciate more recommendations on integration testing for jobs that orchestrate Lakeflow pipelines with streaming tables as targets. Any guidance or patterns for testing streaming pipeline jobs safely and reproducibly—such as isolating streaming state and schemas for each test—would be extremely useful.</P><P>Thanks again for the excellent content!</P> Tue, 03 Mar 2026 15:32:13 GMT https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/149712#M944 de_ig 2026-03-03T15:32:13Z https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/151177#M950 <P>Great question! We've heard this from several folks. We're planning a Part 2 of the blog focused specifically on integration testing SDP pipelines.</P> <P>A quick preview of the key insights we'll cover:</P> <UL> <LI>Fresh pipeline per test is the way to go. Reusing a deployed pipeline across tests breaks because the pipeline's internal metadata retains references to previously dropped ephemeral schemas. Instead, read the deployed pipeline's spec and create a new isolated pipeline instance (via ws.pipelines.create() from the Databricks SDK for Python) targeting each test's ephemeral schema.</LI> <LI>Watch out: the SDK's wait_get_pipeline_idle() returns on failure too - always verify the latest update status is COMPLETED before asserting on table contents.</LI> </UL> <P>Stay tuned!</P> Wed, 18 Mar 2026 08:56:44 GMT https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/151177#M950 Taras_Chaikovsk 2026-03-18T08:56:44Z https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/156129#M1041 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/35260">@Taras_Chaikovsk</a>&nbsp;is the part 2 already out?</P> Tue, 05 May 2026 05:57:16 GMT https://community.databricks.com/t5/technical-blog/integration-testing-for-lakeflow-jobs-with-pytest-and-databricks/bc-p/156129#M1041 yit337 2026-05-05T05:57:16Z