topic Re: Can I give different git branches in the same repo for different tasks in a data bricks workflow in Get Started Discussions

Can I give different git branches in the same repo for different tasks in a data bricks workflow

DBricksNewbie — Mon, 21 Oct 2024 03:20:51 GMT

I have 2 tasks (T1 &T2) that run in branch B1 of Repo1.

I have created a new task (depends on T2 ) which points to a different branch B2 of same Repo1.

Is it possible to run them in the same workflow pipeline? When I tried to set this up, databricks changed the entire pipeline repo branch to B2 instead

Re: Can I give different git branches in the same repo for different tasks in a data bricks workflow

Walter_C — Mon, 04 Nov 2024 20:59:22 GMT

As per docs, this is currently not available:

All tasks in a job must reference the same commit in the remote repository. You must specify only one of the following for a job that uses a remote repository:

branch: The name of the branch, for example, main.
tag: The tag’s name, for example, release-1.0.0.
commit: The hash of a specific commit, for example, e0056d01.

Reference: https://docs.databricks.com/en/jobs/configure-job.html#git:~:text=branch%3A%20The%20name%20of%20the%20branch%2C%20for%20example%2C%20main.

Re: Can I give different git branches in the same repo for different tasks in a data bricks workflow

DBricksNewbie — Tue, 05 Nov 2024 14:57:39 GMT

I was to able to find a workaround. Created separate jobs for those that need to be in different branch (testing tasks) and then ran all of them from a new job.