Hi @Benedetta, In Databricks, parallel runs involve executing multiple tasks concurrently.
Here are some key points related to job IDs and parallel execution:
Job IDs in Parallel Runs:
- Previously, Databricks returned job IDs for parallel runs. These IDs helped identify which link corresponded to the output of a specific task.
- However, recent changes might have impacted the availability of these job IDs. Letโs explore alternative approaches to achieve your goal.
Matching Links to Outputs:
- To match up links with specific outputs, consider the following strategies:
- Task-Level Parameters: Configure parameters at the job level that are passed to all tasks. These parameters can include information relevant to identifying outputs.
- Dynamic Value References: Use dynamic value references to share context between jobs and tasks. These references allow you to pass information dynamically.
- Tags: Add tags (labels or key-value attributes) to your job. Tags can help you filter and organize jobs based on specific criteria.
Cluster Execution Contexts:
- Databricks imposes a hard limit of 145 active execution contexts on a cluster. This ensures the cluster isnโt overloaded with too many parallel threads competing for resources.
- If you have more than 145 parallel jobs, consider creating a new cluster to manage the load effectively.
Editing Job Configuration:
- To modify job settings, follow these steps:
- Click Workflows in the sidebar.
- Locate the job name in the Name column and click it.
- You can adjust various configurations in the side panel, including triggers, compute settings, notifications, and maximum concurrent runs.
- If job access control is enabled, you can also edit permissions.
If you encounter any specific challenges, please ask for further assistance! ๐
