cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Job Timeout after 20 minutes

ibrahim21124
New Contributor III

Hello,

I have created a job with no timeout-seconds provided. But I am getting Error: Timed out within 20 minutes. I am running the below commands using Bash@3 task in ADO Pipeline yaml file. The code for the same is given below

task: Bash@3
  timeoutInMinutes: '0'
  displayName: 'Run Databricks Tests Notebook'
  inputs:
    targetType: 'filePath'
    filePath: './ci/createjob.sh'
    arguments: '$(cluster-id) $(unit-folder) $(component-folder) $(Build.BuildId) $(Build.BinariesDirectory)'
 

This is the json I am passing to create a job.

databricks jobs create --json {
    "name": "Test Job",
    "timeout_seconds": 0,
    "tasks":
    [
        {
            "task_key": "abc",
            "existing_cluster_id": "cluster-id",      
            "notebook_task": {
                "notebook_path": "<path-to-notebook>",
                "source": "WORKSPACE"
                }
        }
}

Also I am starting the job with databricks run-now CLI Command 
databricks jobs run-now --json '{
  "job_id": '$JOB_ID',
  "notebook_params": {
      "param": "value"
  }
}'
 
Is there a way I can check where is this failing after 20 minutes?
1 ACCEPTED SOLUTION

Accepted Solutions

Hey Fatma,

Thanks for your response. I got the solution. Actually when using Databricks CLI and triggering run-now from CLI it is having default timeout of 20 minutes. I have changed that and now it is working as expected.

Thanks,

Ibrahim

View solution in original post

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @ibrahim21124 ,

  • First, ensure that the job timeout is correctly set to 0 seconds in your job configuration.
  • If you’re using the Databricks REST API to create the job, make sure the "timeout_seconds" field is set to 0.
  • Double-check the job configuration to ensure there are no conflicting settings that might override the timeout.
  1.  

Hey Fatma,

Thanks for your response. I got the solution. Actually when using Databricks CLI and triggering run-now from CLI it is having default timeout of 20 minutes. I have changed that and now it is working as expected.

Thanks,

Ibrahim

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!