Can't enable CLI 2.1 on CI
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-15-2022 11:26 AM
Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?
Error:
Resetting Databricks Job with job_id 1036...
WARN: Your CLI is configured to use Jobs API 2.0. In order to use the latest Jobs features please upgrade to 2.1: 'databricks jobs configure --version=2.1'. Future versions of this CLI will default to the new Jobs API. Learn more at https://docs.databricks.com/dev-tools/cli/jobs-cli.html
YML
image: python:3.6-alpine
stages:
- build
- deploy
before_script:
- apk add jq
- echo "Installing Databricks CLI..."
- pip3 install databricks-cli
- pip3 show databricks-cli
- echo "Configuring Databricks profile..."
- echo "[DEFAULT]" > ~/.databrickscfg
- echo "host = ${DATABRICKS_HOST}" >> ~/.databrickscfg
- echo "token = ${DATABRICKS_TOKEN}" >> ~/.databrickscfg
- databricks jobs configure --version=2.1
- source ci/utils.sh
build:
stage: build
only:
- master
script:
- echo "Compiling Python package in EGG distribution..."
- python3 setup.py bdist_egg
- export PACKAGE_VERSION=$(python setup.py --version)
- echo "Pushing compiled library to DBFS..."
- dbfs cp --overwrite dist/${CI_PROJECT_NAME}-${PACKAGE_VERSION}-py3.6.egg dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
deploy:
stage: deploy
only:
- master
script:
- databricks jobs configure --version=2.1
- echo "Pushing compiled library to DBFS..."
- dbfs cp --overwrite dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg dbfs:/FileStore/eggs/${CI_PROJECT_NAME}.egg
- dbfs rm dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
- upload_process_scripts ${DATABRICKS_MAIN_SCRIPT_PATH} ${CI_PROJECT_NAME}
- create_jobs ${DATABRICKS_JOBS_PATH}
when: manual
Thanks!
- Labels:
-
Ci
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2022 11:20 AM
Hello, @Alejandro Martinez - My name is Piper, and I'm a moderator for Databricks. I apologize that it has taken so long to respond to you. 😞
We will give it a bit longer to see what the community has to say before we come back around if we need to.
Thanks for your patience.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2024 08:31 AM
any updated on this one, i am having a similar issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-05-2025 01:04 PM
I got to solve this by downgrading the Databricks runtime to 13.3 and had the below commands for optimization and it worked well in my case.