Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?
Error:
Resetting Databricks Job with job_id 1036...
WARN: Your CLI is configured to use Jobs API 2.0. In order to use the latest Jobs features please upgrade to 2.1: 'databricks jobs configure --version=2.1'. Future versions of this CLI will default to the new Jobs API. Learn more at https://docs.databricks.com/dev-tools/cli/jobs-cli.html
โ
YML
image: python:3.6-alpine
stages:
- build
- deploy
โ
before_script:
- apk add jq
- echo "Installing Databricks CLI..."
- pip3 install databricks-cli
- pip3 show databricks-cli
- echo "Configuring Databricks profile..."
- echo "[DEFAULT]" > ~/.databrickscfg
- echo "host = ${DATABRICKS_HOST}" >> ~/.databrickscfg
- echo "token = ${DATABRICKS_TOKEN}" >> ~/.databrickscfg
- databricks jobs configure --version=2.1
- source ci/utils.sh
โ
build:
stage: build
only:
- master
script:
- echo "Compiling Python package in EGG distribution..."
- python3 setup.py bdist_egg
- export PACKAGE_VERSION=$(python setup.py --version)
- echo "Pushing compiled library to DBFS..."
- dbfs cp --overwrite dist/${CI_PROJECT_NAME}-${PACKAGE_VERSION}-py3.6.egg dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
โ
deploy:
stage: deploy
only:
- master
script:
- databricks jobs configure --version=2.1
- echo "Pushing compiled library to DBFS..."
- dbfs cp --overwrite dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg dbfs:/FileStore/eggs/${CI_PROJECT_NAME}.egg
- dbfs rm dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
- upload_process_scripts ${DATABRICKS_MAIN_SCRIPT_PATH} ${CI_PROJECT_NAME}
- create_jobs ${DATABRICKS_JOBS_PATH}
when: manual
โ
Thanks!
