Databricks Community

DouglasMoore · ‎05-07-2024

Setup UCX from Databricks web terminal

For cases when your desktop or laptop will not support the databricks labs install ucx technical requirements such as Python 3.10, administrative access, python package access or network access but your Databricks cluster will have the Internet access requirements same as described in the main README.md document.

Start an Assigned (single user) interactive cluster
Open the web terminal (Cluster -> Apps -> Web Terminal), expand to full page for better readability

Install and setup needed UCX dependencies:

apt-get update
apt install -y python3.10-venv
unset DATABRICKS_RUNTIME_VERSION

Download and install Databricks cli (not needed for DBR 15.1 and above):

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Set PATH variable to point to new install (not needed for DBR 15.1 and above):

export PATH=/usr/local/bin:$PATH

Verify Databricks cli (Should be greater than 0.219.0):

which databricks
databricks --version

Configure Databricks cli to authenticate to correct workspace, add host name, PAT token (not needed for DBR 15.1 and above):

export DATABRICKS_HOST=https://<hostname>
export DATABRICKS_TOKEN=dapideadbeefdeadbeefdeadbeef

Verify configuration, connection and Databricks labs:

databricks auth env
databricks clusters list
databricks labs list

Install UCX, do not select options to open config.yml or README.py in a browser:

databricks labs install ucx

To Upgrade UCX, do not select options to open config.yml or README.py in a browser:

databricks labs uninstall ucx
databricks labs install ucx

Yeshwanth · ‎05-07-2024

@DouglasMoore thank you for sharing this! 🙌

DouglasMoore · ‎07-22-2024

Please note, With respect to the Databricks CLI, the Web Terminal shell behaves differently with DBR 15.1 and above. The call outs are inline above the relevant commands.

Sourabh_13 · ‎07-24-2024

Hi @DouglasMoore,

I would like to provide some clarification regarding the commands mentioned in your post about DBR 15.1 or above. Contrary to what was stated, these commands are indeed necessary. Recently, when attempting to install UCX via the web terminal on my cluster, I encountered specific issues that were resolved by executing these commands.

The recommended flow is as follows:

1. Begin by installing the required UCX dependencies and setting up the Python virtual environment:

apt-get update
apt install -y python3.10-venv
unset DATABRICKS_RUNTIME_VERSION

2. Next, download and install the Databricks CLI:

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

3. Configure the Databricks CLI to authenticate with the correct workspace, adding the hostname and PAT token:

export DATABRICKS_HOST=https://<hostname>
export DATABRICKS_TOKEN=<PAT>

4. Finally, proceed to install UCX:

databricks labs install ucx

Following these steps ensures the correct installation of UCX. It's important to note that failure to provide the Databricks hostname and PAT token, even with DBR 15.1 or above, can lead to installation failures on Azure with AWS-linked validation errors.