Issue with Creating and Running Databricks Jobs with new databricks cli v0.214.0

[Kishor]

Hi Databricks Support,

I'm encountering an issue with creating and running jobs on Databricks. Here are the details:

Problem Description:
When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs using the old Databricks CLI version 0.17.8), I encountered an error. Although job creation was successful, running the job resulted in the following error: "Error: No task is specified."

Steps Taken:
Created a job using the old JSON file with the command:

databricks jobs create --json @sample.json

Job creation was successful, but running the job resulted in an error.

Updated the JSON file based on a sample from the Databricks GitHub repository and tried creating and running the job again. This time, both job creation and job run commands worked fine.

However, I encountered another error when attempting to retrieve the run output:

databricks jobs get-run-output 89359307425900

The error message received was: "Error: Retrieving the output of runs with multiple tasks is not supported. Please retrieve the output of each individual task run instead."

JSON Details:

Initial JSON (used for job creation with old Databricks CLI v0.17.8):

{
"libraries": [],
"name": "nabu-sparkbot-custom-code-arg-test",
"max_concurrent_runs": 1,
"timeout_seconds": 259200,
"access_control_list": [],
"notebook_task": {
"notebook_path": "/dbfs/tmp/sample/sample.py"
},
"new_cluster": {
"spark_version": "10.4.x-scala2.12",
"node_type_id": "Standard_DS3_v2",
"enable_elastic_disk": true,
"num_workers": 2,
"spark_conf": {
"spark.dynamicAllocation.enabled": "false"
},
"runtime_engine": "STANDARD"
}
}

Updated JSON (used for successful job creation and run):

{
"name": "nabu-sparkbot-custom-code-arg-test",
"tasks": [
{
"job_cluster_key": "create-job-without-workers-cluster",
"task_key": "create-job-without-workers-cluster1",
"libraries": [],
"max_concurrent_runs": 1,
"timeout_seconds": 259200,
"notebook_task": {
"notebook_path": "/dbfs/tmp/sample/sample.py"
}
}
],
"job_clusters": [
{
"job_cluster_key": "create-job-without-workers-cluster",
"new_cluster": {
"spark_version": "10.4.x-scala2.12",
"node_type_id": "Standard_DS3_v2",
"enable_elastic_disk": true,
"num_workers": 2,
"spark_conf": {
"spark.dynamicAllocation.enabled": "false"
},
"runtime_engine": "STANDARD"
}
}
]
}

Request:
Could you please assist in resolving the issue with job creation and retrieval of run output? Additionally, any guidance on creating and running individual task runs would be greatly appreciated.

Thank you for your assistance.

Best regards,
kishor.chintanpalli@modak.com

[Kaniz_Fatma]

Hi @Kishor, I’m sorry to hear that you’re having trouble with Databricks job creation and retrieval of run output.

Issue 1: “Error: No task is specified.” This error typically occurs when the JSON file used for job ...¹. In your initial JSON, you have specified a notebook_task, which should be sufficient. However, it seems that the Databricks CLI version you are using might be expecting a different format or additional information.

One possible solution could be to ensure that your JSON file is correctly formatted and includes all necessary information. For example, you might need to specify spark_jar_task, spark_python_task or notebook_task².

Issue 2: “Error: Retrieving the output of runs with multiple tasks is not supported. Please retrieve...³. Instead, you should retrieve the output of each individual task run⁴.

To retrieve the output of individual task runs, you can use the get-run command with the run-id of t...⁴. This will show the tasks array containing all the task runs⁴.

Creating and Running Individual Task Runs To create and run individual task runs in Databricks, you ...¹⁵:

1. Click Workflows in the sidebar and click +.
2. In the sidebar, click New and select Job. The Tasks tab appears with the create task dialog along with the Job details side panel containing job-level settings.
3. Replace New Job… with your job name. Enter a name for the task in the Task name field.
4. In the Type drop-down menu, select the type of task to run.
5. Configure the cluster where the task runs.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

[Kishor]

Hi @Kaniz_Fatma ,

Thanks for the reference links for the solution.

I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384  GitHub. By using the get-run API, I was able to retrieve the running status of my job along with a detailed description.

Following the guidance and using the `get-run option, I managed to get the job running status and description successfully.