topic Re: What is a good way to implement unit tests using github actions for databricks? in Data Engineering

What is a good way to implement unit tests using github actions for databricks?

Ullsokk — Fri, 10 Feb 2023 15:13:16 GMT

I am trying to use a git template for unit tests on a databricks project. The framework uses pylint, pytest and black to check the code. But I am having a lot of trouble getting the github actions vm to run the code without issues. I have had issues with spark and hadoop versions, packages, java errors. Is it even a good idea to have github actions fire up a vm that is identical to my databricks environment and test the code? Is there another more sensible approach, such using the run-notebook action instead? Should I look into some completely different framework for doing unit tests?

Re: What is a good way to implement unit tests using github actions for databricks?

Anonymous — Thu, 09 Mar 2023 02:36:20 GMT

@Stian Arntsen :

Using GitHub Actions can be a tedious way to run unit tests for the reasons you have mentioned above.

You can use an another approach to do this. Use the run-notebook action to run your unit tests. This action allows you to specify a Databricks notebook to run and the cluster to run it on. You can configure the notebook to execute your unit tests and report the results back to GitHub. This approach may be simpler and more reliable than trying to replicate your entire Databricks environment on a GitHub Actions VM

Re: What is a good way to implement unit tests using github actions for databricks?

Anonymous — Sun, 09 Apr 2023 07:00:52 GMT

Hi @Stian Arntsen

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!