topic Re: Setup UCX (Databricks CLI) from Databricks web terminal in Data Governance

Setup UCX (Databricks CLI) from Databricks web terminal

DouglasMoore — Wed, 08 May 2024 17:20:35 GMT

Setup UCX from Databricks web terminal

For cases when your desktop or laptop will not support the databricks labs install ucx technical requirements such as Python 3.10, administrative access, python package access or network access but your Databricks cluster will have the Internet access requirements same as described in the main README.md document.

Start an Assigned (single user) interactive cluster
Open the web terminal (Cluster -> Apps -> Web Terminal), expand to full page for better readability

Install and setup needed UCX dependencies:

apt-get update
apt install -y python3.10-venv
unset DATABRICKS_RUNTIME_VERSION

Download and install Databricks cli (not needed for DBR 15.1 and above):

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Set PATH variable to point to new install (not needed for DBR 15.1 and above):

export PATH=/usr/local/bin:$PATH

Verify Databricks cli (Should be greater than 0.219.0):

which databricks
databricks --version

Configure Databricks cli to authenticate to correct workspace, add host name, PAT token (not needed for DBR 15.1 and above):

export DATABRICKS_HOST=https://<hostname>
export DATABRICKS_TOKEN=dapideadbeefdeadbeefdeadbeef

Verify configuration, connection and Databricks labs:

databricks auth env
databricks clusters list
databricks labs list

Install UCX, do not select options to open config.yml or README.py in a browser:

databricks labs install ucx

To Upgrade UCX, do not select options to open config.yml or README.py in a browser:

databricks labs uninstall ucx
databricks labs install ucx

Re: Setup UCX (Databricks CLI) from Databricks web terminal

Yeshwanth — Wed, 08 May 2024 06:46:54 GMT

@DouglasMoore thank you for sharing this! 🙌

Re: Setup UCX (Databricks CLI) from Databricks web terminal

DouglasMoore — Mon, 22 Jul 2024 12:27:59 GMT

Please note, With respect to the Databricks CLI, the Web Terminal shell behaves differently with DBR 15.1 and above. The call outs are inline above the relevant commands.

Re: Setup UCX (Databricks CLI) from Databricks web terminal

Sourabh_13 — Wed, 24 Jul 2024 11:56:04 GMT

Hi @DouglasMoore,

I would like to provide some clarification regarding the commands mentioned in your post about DBR 15.1 or above. Contrary to what was stated, these commands are indeed necessary. Recently, when attempting to install UCX via the web terminal on my cluster, I encountered specific issues that were resolved by executing these commands.

The recommended flow is as follows:

1. Begin by installing the required UCX dependencies and setting up the Python virtual environment:

apt-get update
apt install -y python3.10-venv
unset DATABRICKS_RUNTIME_VERSION

2. Next, download and install the Databricks CLI:

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

3. Configure the Databricks CLI to authenticate with the correct workspace, adding the hostname and PAT token:

export DATABRICKS_HOST=https://<hostname>
export DATABRICKS_TOKEN=<PAT>

4. Finally, proceed to install UCX:

databricks labs install ucx

Following these steps ensures the correct installation of UCX. It's important to note that failure to provide the Databricks hostname and PAT token, even with DBR 15.1 or above, can lead to installation failures on Azure with AWS-linked validation errors.

Re: Setup UCX (Databricks CLI) from Databricks web terminal

DouglasMoore — Wed, 24 Jul 2024 16:19:37 GMT

@Sourabh_13 Thanks for your testing and feedback. What was your exact version of the Databricks Runtime (DBR) ?

Re: Setup UCX (Databricks CLI) from Databricks web terminal

Sourabh_13 — Thu, 25 Jul 2024 10:50:31 GMT

@DouglasMoore DBR version that i used for this testing was 15.3.

Re: Setup UCX (Databricks CLI) from Databricks web terminal

andres_garcia — Tue, 21 Jan 2025 15:06:21 GMT

I’ve found that using DBR 15.4, Personal Compute, and a Single Node is the most effective setup for installing UCX via web terminal.

Note:

The Databricks CLI is the recommended and most efficient option for managing the UCX installation. Always try the CLI first for a streamlined and reliable experience.

This web terminal workaround can be used if you’re facing restrictions such as limitations on software installation, firewall rules blocking necessary sites, or other constraints. Use it only in such scenarios where the CLI is not feasible.

Follow these steps for the installation:

Step 1: Install Required Dependencies and Set Up Python Virtual Environment

Run the following commands to ensure your environment is ready:

sudo apt update && sudo apt upgrade -y && sudo apt --fix-broken install -y && sudo apt install -y python3.10-venv

Step 2: Unset the Runtime Version Environment Variable

unset DATABRICKS_RUNTIME_VERSION

Step 3: Install the Databricks CLI (Optional for DBR 15.4 and above)

If you encounter any issues calling the CLI, install it using this command:

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Step 4: Install UCX

Finally, install UCX with the following command (make sure you’re on DBR 15.4 or later) :

databricks labs install ucx