topic Re: When developing a Delta Live Table, can I test my code in the notebook? in Data Engineering

When developing a Delta Live Table, can I test my code in the notebook?

User16826992666 — Fri, 25 Jun 2021 19:09:57 GMT

Just wondering about the dev flow when building Delta Live Tables. I write my code in the notebook so it would be useful to be able to test it out from within that environment.

Re: When developing a Delta Live Table, can I test my code in the notebook?

Srikanth_Gupta_ — Fri, 25 Jun 2021 19:16:15 GMT

I don't think Delta Live Table code is interactive, we need to create Pipeline and test the code.

Re: When developing a Delta Live Table, can I test my code in the notebook?

sajith_appukutt — Fri, 25 Jun 2021 22:11:00 GMT

By test , did you mean running them from notebooks ( as against an IDE ) or, unit testing from notebooks.

If it is the first, checkout example notebooks. If it is the latter you could abstract out transformations into separate functions ( that accepts a dataframe and returns a dataframe) and have them unit tested using standard unit testing practices.

Re: When developing a Delta Live Table, can I test my code in the notebook?

aladda — Fri, 25 Jun 2021 22:34:49 GMT

The delta live tables code in a notebook is essentially a template for how you want your data transformed. The Pipelines feature is the execution/operationlization component for that notebook.

With respect to being able to test the notebook, one option could be to setup up a Dev/Test Pipeline that lands the data in a dev/test target database run validation queries either using a different databricks notebook or Databricks SQL and then push it to production with a different pipeline that send data to a production target database

Re: When developing a Delta Live Table, can I test my code in the notebook?

martinez — Mon, 17 Jul 2023 08:37:03 GMT

Hey 🤗

As a fellow developer, I can definitely relate to the question User16826992666 has raised regarding testing code in a notebook when developing a Delta Live Table.

In my experience, notebooks are a fantastic tool for exploratory data analysis and prototyping code. When it comes to testing your code for Delta Live Tables, you can certainly leverage the notebook environment to your advantage. It allows you to execute code cells and observe the results, making it convenient for iterative development.

I would recommend using the notebook to validate your code logic, verify data transformations, and ensure that your Delta Live Table behaves as expected. You can simulate the expected behavior of your live table by creating sample data or using subsets of your actual data. Tutorial here: https://andersenlab.com/services/quality-assurance-services

However, keep in mind that notebooks may not fully replicate the production environment, especially when it comes to handling large-scale data or simulating real-time data updates. So, while the notebook is great for initial testing and debugging, it's essential to conduct thorough testing and performance evaluation in a dedicated development or staging environment before deploying your Delta Live Table to production.

Happy coding and testing! ✍️

Re: When developing a Delta Live Table, can I test my code in the notebook?

Lakshay — Mon, 17 Jul 2023 12:20:23 GMT

You need to create a DLT pipeline to test the code.