In my opinion as databricks is cloud system build on top of spark so the best option is to have environments in cloud and run tests there. So basically all tests will be run via Jobs API and you use Repos. Of course you can still use Docker in all 3 environments:

• developer environment,
• build pipeline environment (where we deploy and run test after commit/pull request)
• release pipeline (production)

Tests can be something like run SQL queries and compare results etc (returned via Jobs). I thought also about implementing SQL alarms for testing but they are still in preview.

I also think that as Repos are quite new, we can expect more improvements with CI/CD pipelines as current way is far from perfection 🙂