Databricks Community

alejandrofm · ‎02-15-2022

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?

Error:

Resetting Databricks Job with job_id 1036...

WARN: Your CLI is configured to use Jobs API 2.0. In order to use the latest Jobs features please upgrade to 2.1: 'databricks jobs configure --version=2.1'. Future versions of this CLI will default to the new Jobs API. Learn more at https://docs.databricks.com/dev-tools/cli/jobs-cli.html

YML

image: python:3.6-alpine
 
stages:
 
 - build
 
 - deploy
 

 
before_script:
 
   - apk add jq
 
   - echo "Installing Databricks CLI..." 
 
   - pip3 install databricks-cli
 
   - pip3 show databricks-cli
 
   - echo "Configuring Databricks profile..."
 
   - echo "[DEFAULT]" > ~/.databrickscfg
 
   - echo "host = ${DATABRICKS_HOST}" >> ~/.databrickscfg
 
   - echo "token = ${DATABRICKS_TOKEN}" >> ~/.databrickscfg
 
   - databricks jobs configure --version=2.1
 
   - source ci/utils.sh
 

 
build:
 
   stage: build
 
   only:
 
       - master
 
   script:
 
       - echo "Compiling Python package in EGG distribution..."
 
       - python3 setup.py bdist_egg
 
       - export PACKAGE_VERSION=$(python setup.py --version)
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dist/${CI_PROJECT_NAME}-${PACKAGE_VERSION}-py3.6.egg dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 

 
deploy:
 
   stage: deploy
 
   only:
 
       - master
 
   script:
 
       - databricks jobs configure --version=2.1
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg dbfs:/FileStore/eggs/${CI_PROJECT_NAME}.egg
 
       - dbfs rm dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 
       - upload_process_scripts ${DATABRICKS_MAIN_SCRIPT_PATH} ${CI_PROJECT_NAME}
 
       - create_jobs ${DATABRICKS_JOBS_PATH}
 
   when: manual

Thanks!