The way we do this is to package as much re-usable code up into a common library as possible and then test it to within an inch of it's life with unit tests (I tend to use unittest for lower barrier to entry, but which ever framework works best for you). This includes putting any user defined functions or Spark API functions through unit tests with Spark running locally. We then have build pipelines in Azure DevOps (though this works with Github actions as well) that lints, tests, builds, and then deploys the library to Databricks where it can be pulled in by the notebooks. Ideally leaving the notebooks to read in and write out data frames, but with the bulk of the work in the libraries.

That's assuming use of notebooks, and not submitting whole jobs as jar/wheel files.

I've done a blog post on unit testing PySpark libraries which is online for anyone to read.

For linting I normally pull in the following libraries

flake8

pep8-naming

flake8-docstrings

flake8-bandit

flake8-eradicate

These then lint, ensure naming conventions, check that docstrings have been created properly, check for common security issues, and identifies commented out code. I get pretty brutal with the builds and if any tests fail, linting fails, or coverage drops below 90% then then entire build fails.

I also wrote one on auto-generating documentation and publishing to an Azure Static Web App (with authentication) so anyone on the internal team can use it.

Hope that all helps