I have a process in DBX/DAB and I am using Service Principal for generating a token for reaching the artifacts feed, for security this token lasts 1 hour.

import requests

YOUR_AZURE_TENANT_ID = ...
YOUR_SERVICE_PRINCIPAL_CLIENT_ID = ...
YOUR_SECRET_SCOPE = ...
YOUR_SECRET_KEY = ...
SCOPE = ... # Scope for Azure DevOps Services API

url = f'https://login.microsoftonline.com/{YOUR_AZURE_TENANT_ID}/oauth2/v2.0/token'
 
payload = {'grant_type' : 'client_credentials',
           'client_id' : YOUR_SERVICE_PRINCIPAL_CLIENT_ID,
           'client_secret': dbutils.secrets.get(scope=YOUR_SECRET_SCOPE, key=YOUR_SECRET_KEY),
           'scope': SCOPE}
 
files = [
    ...
]
 
headers = {
    ...
}

response = requests.request("POST", url, headers=headers, data = payload, files = files)

# get the "access_token" from the response
access_token = response.json().get("access_token")

print(access_token)

I want to have a workflow that runs the token generation (which is the above code) and then run the original workflow.

1. Generate Token generates the token and leaves it in a location defined in the .yaml
2. The main workflow task in its init script gets the content of the file and sets the PIP_EXTRA_INDEX_URL env var.

The thing is that I made a workflow that does that, but the original workflow starts searching for libraries instead of waiting for setting the token of the first workflow.

When I achieve this independently it works (first generate the token then the original workflow runs), but it is cost inneficient, because I need to renew the token each hour.

  workflows:
    - name: dev-workflow-process

      job_clusters:
        - job_cluster_key: dev
          new_cluster:
            init_scripts:
              - workspace:
                  destination: "/Workspace/Shared/SP-LIB-INSTALLATION/init_pip_extra_index_url.sh"
            spark_version: "11.3.x-cpu-ml-scala2.12"
            driver_node_type_id: "Standard_F8"
            node_type_id: "Standard_F8"
            num_workers: 2
            spark_env_vars:
              TOKEN_FILENAME: "pip_token"

        - job_cluster_key: "generate-token"
          new_cluster:
            spark_version: "11.3.x-cpu-ml-scala2.12"
            driver_node_type_id: "Standard_F8"
            node_type_id: "Standard_F8"
            num_workers: 1
      tasks:
        - task_key: "generate-token"
          job_cluster_key: "generate-token"
          notebook_task:
            notebook_path: "/Workspace/Shared/SP-LIB-INSTALLATION/GenerateAndSaveToken"  # Updated path
            base_parameters:
              TOKEN_FILENAME: "pip_token"  # Specify the desired token file name here

        - task_key: "main-task"
          depends_on:
          - task_key: "generate-token"
          job_cluster_key: !? $.env
          python_wheel_task:
              package_name: "dev-workflow-process"
              entry_point: "entrypoint"
              parameters:
              - "--conf-file"
              - "file:fuse://conf/tasks/main_task_config.yml"
build:
  python: "poetry"

Is there a way to force the second workflow to wait for the token to be generated?