Hi @maikel 

1. Databricks Connect (Best fit for your situation)
This is likely your best path. It lets you run Spark code locally or in CI against a real Databricks cluster/serverless compute, meaning:

 - Real Spark behavior, no mocking
 - Tests run from your local machine or CI pipeline (GitHub Actions, Azure DevOps, etc.)
 - You write standard pytest tests
 - Serverless compute is supported as of Databricks Connect v2 (DBR 13+)
Your code and tests run locally, but all actual Spark execution happens on Databricks. No mocking, real Delta, real Unity Catalog.
Before writing any tests, verify your connection works:

2. Nutter (Databricks-native notebook testing)
If your logic is tightly coupled to notebooks, Nutter is a framework by Microsoft specifically for testing Databricks notebooks. It runs notebooks as tests inside the Databricks environment.
Good if you want to test notebook-level behavior, but less clean for pure function unit tests.

Regards,
LR