Can't enable CLI 2.1 on CI

alejandrofm
Valued Contributor

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?

Error:

Resetting Databricks Job with job_id 1036...

WARN: Your CLI is configured to use Jobs API 2.0. In order to use the latest Jobs features please upgrade to 2.1: 'databricks jobs configure --version=2.1'. Future versions of this CLI will default to the new Jobs API. Learn more at https://docs.databricks.com/dev-tools/cli/jobs-cli.html

YML

image: python:3.6-alpine
 
stages:
 
 - build
 
 - deploy
 
​
 
before_script:
 
   - apk add jq
 
   - echo "Installing Databricks CLI..." 
 
   - pip3 install databricks-cli
 
   - pip3 show databricks-cli
 
   - echo "Configuring Databricks profile..."
 
   - echo "[DEFAULT]" > ~/.databrickscfg
 
   - echo "host = ${DATABRICKS_HOST}" >> ~/.databrickscfg
 
   - echo "token = ${DATABRICKS_TOKEN}" >> ~/.databrickscfg
 
   - databricks jobs configure --version=2.1
 
   - source ci/utils.sh
 
​
 
build:
 
   stage: build
 
   only:
 
       - master
 
   script:
 
       - echo "Compiling Python package in EGG distribution..."
 
       - python3 setup.py bdist_egg
 
       - export PACKAGE_VERSION=$(python setup.py --version)
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dist/${CI_PROJECT_NAME}-${PACKAGE_VERSION}-py3.6.egg dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 
​
 
deploy:
 
   stage: deploy
 
   only:
 
       - master
 
   script:
 
       - databricks jobs configure --version=2.1
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg dbfs:/FileStore/eggs/${CI_PROJECT_NAME}.egg
 
       - dbfs rm dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 
       - upload_process_scripts ${DATABRICKS_MAIN_SCRIPT_PATH} ${CI_PROJECT_NAME}
 
       - create_jobs ${DATABRICKS_JOBS_PATH}
 
   when: manual
 
​

Thanks!

  • Ci

Anonymous
Not applicable

Hello, @Alejandro Martinez​ - My name is Piper, and I'm a moderator for Databricks. I apologize that it has taken so long to respond to you. 😞

We will give it a bit longer to see what the community has to say before we come back around if we need to.

Thanks for your patience.

karthik-kandiko
New Contributor II

any updated on this one, i am having a similar issue 

karthik-kandiko
New Contributor II

I got to solve this by downgrading the Databricks runtime to 13.3 and had the below commands for optimization and it worked well in my case.

spark.conf.set("spark.sql.shuffle.partitions", "200")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")