Can't enable CLI 2.1 on CI

alejandrofm — Tue, 15 Feb 2022 19:26:04 GMT

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?

Error:

Resetting Databricks Job with job_id 1036...

WARN: Your CLI is configured to use Jobs API 2.0. In order to use the latest Jobs features please upgrade to 2.1: 'databricks jobs configure --version=2.1'. Future versions of this CLI will default to the new Jobs API. Learn more at https://docs.databricks.com/dev-tools/cli/jobs-cli.html

YML

image: python:3.6-alpine
 
stages:
 
 - build
 
 - deploy
 

 
before_script:
 
   - apk add jq
 
   - echo "Installing Databricks CLI..." 
 
   - pip3 install databricks-cli
 
   - pip3 show databricks-cli
 
   - echo "Configuring Databricks profile..."
 
   - echo "[DEFAULT]" > ~/.databrickscfg
 
   - echo "host = ${DATABRICKS_HOST}" >> ~/.databrickscfg
 
   - echo "token = ${DATABRICKS_TOKEN}" >> ~/.databrickscfg
 
   - databricks jobs configure --version=2.1
 
   - source ci/utils.sh
 

 
build:
 
   stage: build
 
   only:
 
       - master
 
   script:
 
       - echo "Compiling Python package in EGG distribution..."
 
       - python3 setup.py bdist_egg
 
       - export PACKAGE_VERSION=$(python setup.py --version)
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dist/${CI_PROJECT_NAME}-${PACKAGE_VERSION}-py3.6.egg dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 

 
deploy:
 
   stage: deploy
 
   only:
 
       - master
 
   script:
 
       - databricks jobs configure --version=2.1
 
       - echo "Pushing compiled library to DBFS..."
 
       - dbfs cp --overwrite dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg dbfs:/FileStore/eggs/${CI_PROJECT_NAME}.egg
 
       - dbfs rm dbfs:/FileStore/build/${CI_PROJECT_NAME}.egg
 
       - upload_process_scripts ${DATABRICKS_MAIN_SCRIPT_PATH} ${CI_PROJECT_NAME}
 
       - create_jobs ${DATABRICKS_JOBS_PATH}
 
   when: manual

Thanks!

Re: Can't enable CLI 2.1 on CI

Anonymous — Mon, 21 Feb 2022 19:20:22 GMT

Hello, @Alejandro Martinez - My name is Piper, and I'm a moderator for Databricks. I apologize that it has taken so long to respond to you. 😞

We will give it a bit longer to see what the community has to say before we come back around if we need to.

Thanks for your patience.

Re: Can't enable CLI 2.1 on CI

karthik-kandiko — Thu, 10 Oct 2024 15:31:33 GMT

any updated on this one, i am having a similar issue

Re: Can't enable CLI 2.1 on CI

karthik-kandiko — Wed, 05 Feb 2025 21:04:02 GMT

I got to solve this by downgrading the Databricks runtime to 13.3 and had the below commands for optimization and it worked well in my case.

spark.conf.set("spark.sql.shuffle.partitions", "200")

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

topic Re: Can't enable CLI 2.1 on CI in Data Engineering

Can't enable CLI 2.1 on CI

Re: Can't enable CLI 2.1 on CI

Re: Can't enable CLI 2.1 on CI

Re: Can't enable CLI 2.1 on CI