Databricks CLI bundle run multiple jobs

[thibault]

Hi!

I am using bundles to deploy various workflows, and as part of a CI pipeline, I want to run integration tests. They are in notebooks that I deploy and run with databricks CLI bundle with Azure DevOps.

This allows me to run only one job at a time as the command takes only one argument (databricks bundle run --job_test_name)

This means that each job needs to run in sequence and is not really scalable. Is there any workaround so that we could basically do something like this, to run all jobs matching *-test?:

databricks bundle run *-test

[Kaniz_Fatma · Accepted solution]

Hi @thibault, To run multiple jobs matching a pattern like *-test, you can use the Databricks CLI’s bundle commands.

1. First, create a bundle configuration file that specifies the jobs you want to run. You can define multiple jobs in this configuration.
2. In your bundle configuration, define the jobs you want to run. 
3. Use the databricks bundle run command to execute your bundle.

This approach allows you to run multiple jobs concurrently, making your integration tests more scalable. For more details, you can refer to the official Databricks CLI documentation. 

If you encounter any issues or need further assistance, feel free to ask!

[thibault]

Hi @Kaniz_Fatma and thanks for your reply.

I see, so all tests will need to be defined in one config file under one job.

[LettieTaylor]

Thank you so much for the information.