topic Re: Errors Using Selenium/Chromedriver in DataBricks in Data Engineering

Errors Using Selenium/Chromedriver in DataBricks

Gray — Tue, 01 Nov 2022 11:16:36 GMT

Hello,

I’m programming in a notebook and attempting to use the python library Selenium to automate Chrome/chromedriver. I’ve successfully managed to install selenium using

%sh
 pip install selenium

I then attempt the following code, which results in the WebdriverException, copied below.

from selenium import webdriver
driver = webdriver.Chrome()

Error:

WebdriverException: Message: ‘chromedriver’ executable needs to be in PATH. Please see https://chromedriver.chromium.org/home

After troubleshooting the error, I attempted instead to use webdriver-manager to install the instance of chromedriver as follows, whilst also running it headless.

%sh
pip install webdriver-manager

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
 
options = Options()
options.add_argument(“—headless”)
 
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)

This time, I got the following error:

WebdriverException: Message: Service /root/.wdm/drivers/chromedriver/linux64/107.0.5304/chromedriver unexpectedly exited. Status code was: 127

I’ve roamed the internet for a solution, but no matter what I try, my code ends up throwing one of the two WebDriverException errors above.

Does anybody know how I can get selenium running on DataBricks in order to automate Chrome/chromedriver?

Thanks!

Re: Errors Using Selenium/Chromedriver in DataBricks

Hubert-Dudek — Tue, 01 Nov 2022 16:07:36 GMT

Maybe my manual on how to run selenium on Databricks will help:

In the clusters library tab, please install PyPi chromedriver-binary==83.0 (or higher, probably version in the script can also be updated)

Please run the below script from the notebook to create "/databricks/scripts/selenium-install.sh" file.

dbutils.fs.mkdirs("dbfs:/databricks/scripts/")
dbutils.fs.put("/databricks/scripts/selenium-install.sh","""
#!/bin/bash
apt-get update
apt-get install chromium-browser=91.0.4472.101-0ubuntu0.18.04.1 --yes
wget https://chromedriver.storage.googleapis.com/91.0.4472.101/chromedriver_linux64.zip -O /tmp/chromedriver.zip
mkdir /tmp/chromedriver
unzip /tmp/chromedriver.zip -d /tmp/chromedriver/
""", True)
display(dbutils.fs.ls("dbfs:/databricks/scripts/"))

Please add "/databricks/scripts/selenium-install.sh" as starting script - init in cluster config.

Later in the notebook, you can use chrome, as in the below example.

from selenium import webdriver
chrome_driver = '/tmp/chromedriver/chromedriver'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-dev-shm-usage') 
chrome_options.add_argument('--homedir=/dbfs/tmp')
chrome_options.add_argument('--user-data-dir=/dbfs/selenium')
# prefs = {"download.default_directory":"/dbfs/tmp",
#          "download.prompt_for_download":False
# }
# chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)

Re: Errors Using Selenium/Chromedriver in DataBricks

Gray — Thu, 03 Nov 2022 13:10:30 GMT

Hi Hubert,

Thank you for your quick response! I've copied your code across to my notebook. However, when I run the following code

%sh
/dbfs/databricks/scripts/selenium-install.sh

I get the following output

Hit:1 https://repos.azul.com/zulu/deb stable InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '91.0.4472.101-0ubuntu0.18.04.1' for 'chromium-browser' was not found
/dbfs/databricks/scripts/selenium-install.sh: line 5: --yes: command not found
--2022-11-03 13:02:23--  https://chromedriver.storage.googleapis.com/91.0.4472.101/
Resolving chromedriver.storage.googleapis.com (chromedriver.storage.googleapis.com)... 209.85.202.128, 2a00:1450:400b:c01::80
Connecting to chromedriver.storage.googleapis.com (chromedriver.storage.googleapis.com)|209.85.202.128|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-11-03 13:02:24 ERROR 404: Not Found.
 
/dbfs/databricks/scripts/selenium-install.sh: line 7: chromedriver_linux64.zip: command not found
mkdir: invalid option -- 'd'
Try 'mkdir --help' for more information.

And consequently, when I run this code block:

from selenium import webdriver
chrome_driver = '/tmp/chromedriver/chromedriver'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--homedir=/dbfs/tmp')
chrome_options.add_argument('--user-data-dir=/dbfs/selenium')
# prefs = {"download.default_director":"/dbfs/tmp",
#          "download.prompt_for_download":False
# }
# chrome_options.add_experimental_options("prefs",prefs)
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)

I receive the following error:

WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://chromedriver.chromium.org/home

Is this something you can shed some light on for me please?

Thank you for your help!

Re: Errors Using Selenium/Chromedriver in DataBricks

Anonymous — Sun, 06 Nov 2022 07:12:12 GMT

Hi @Henry Gray

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Errors Using Selenium/Chromedriver in DataBricks

Hubert-Dudek — Wed, 09 Nov 2022 14:25:08 GMT

Hi, @Henry Gray . I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL

Re: Errors Using Selenium/Chromedriver in DataBricks

Gray — Fri, 11 Nov 2022 10:04:02 GMT

@Kaniz Fatma @Vidula Khanna @Hubert Dudek

My colleague and I were finally able to get Selenium running in a notebook. Although I can't explain in detail why this solution works, I have attached the source file below.

Hopefully this might help somebody in the future!

Cheers

Re: Errors Using Selenium/Chromedriver in DataBricks

luck_az — Thu, 01 Dec 2022 15:06:33 GMT

Hi @Henry Gray , there is one command in your script, which is. running forever. If i am skipping that command, my chromedriver is not working. [xvfb-run java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -jar selenium-server.jar. Can you please suggest how to proceed?]

Re: Errors Using Selenium/Chromedriver in DataBricks

Gray — Thu, 01 Dec 2022 15:19:00 GMT

Hi,

My colleague and I also found that line started running infinitely. We tinkered with the code and did the following to make it work.

1) Remove the following two portions of code:

%sh
wget https://github.com/SeleniumHQ/selenium/releases/download/selenium-4.1.0/selenium-server-4.1.2.jar
mv selenium-server-4.1.2.jar selenium-server.jar

%sh
sudo apt install xvfb
xvfb-run java -D webdriver.chrome.driver=/usr/bin/chromedriver -jar selenium-server.jar

2) Add the following code to the beginning:

%sh
sudo rm -r /var/lib/apt/lists/* 
sudo apt clean && 
  sudo apt update --fix-missing -y &&
  sudo apt install -y  libmysqlclient21
sudo apt install -y gdal-bin

Additionally, fyi, our runtime version of DataBricks is 0.4 LTS (includes Apache Spark 3.2.1, Scala 2.12).

I'm not sure why this works, but hopefully it will fix your issues.

Cheers!

Re: Errors Using Selenium/Chromedriver in DataBricks

luck_az — Thu, 01 Dec 2022 15:56:14 GMT

Thanks, it worked. Great work.

Re: Errors Using Selenium/Chromedriver in DataBricks

SShiv — Mon, 05 Dec 2022 05:12:32 GMT

I tried this script but got the following response. How do I fix this?

Re: Errors Using Selenium/Chromedriver in DataBricks

luck_az — Mon, 05 Dec 2022 07:27:52 GMT

Hi @Henry Gray , i want to access vpn using selenium in databricks. Do you have any idea , how we can do that ?

Re: Errors Using Selenium/Chromedriver in DataBricks

aa_204 — Thu, 15 Dec 2022 11:36:20 GMT

I also tried the script and am getting similar error. Can anyone please give some resolution for it?

Error in Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/udev_245.4-4ubuntu3.18_amd64.deb and Unable to fetch some archives

Re: Errors Using Selenium/Chromedriver in DataBricks

acristinar — Thu, 22 Dec 2022 17:58:32 GMT

This solution saved my life! Thank you so much for posting it!

Re: Errors Using Selenium/Chromedriver in DataBricks

Anonymous — Tue, 27 Jun 2023 13:20:02 GMT

I had same issue try this:

from this post

%sh
sudo rm -r /var/lib/apt/lists/* 
sudo apt clean && 
   sudo apt update --fix-missing -y

Re: Errors Using Selenium/Chromedriver in DataBricks

Anonymous — Tue, 27 Jun 2023 13:20:43 GMT

I had same issue try this as i answered previous question:

from this post

%sh
sudo rm -r /var/lib/apt/lists/* 
sudo apt clean && 
   sudo apt update --fix-missing -y

Re: Errors Using Selenium/Chromedriver in DataBricks

keithkifo — Thu, 09 Nov 2023 07:10:53 GMT

Hi Gray, I was looking for your script but I don't think you no longer have any file attached to your reply. Would really love your help on this!

Re: Errors Using Selenium/Chromedriver in DataBricks

Kaizen — Thu, 18 Jan 2024 18:34:30 GMT

The attached source file seems to be missing

Also what cluster access type are you running? Shared doesnt let us access the file system since it is protected resulting in error like:
WebDriverException: Message: Can not connect to the Service /databricks/.pyenv/bin/chromedriver

Re: Errors Using Selenium/Chromedriver in DataBricks

dungruoc — Wed, 13 Mar 2024 11:29:30 GMT

Hi @Gray ,

I do not find your attached source file? It might be helpful as I am facing the same issue.

Thanks,

Re: Errors Using Selenium/Chromedriver in DataBricks

Evan_MCK — Wed, 13 Mar 2024 12:53:26 GMT

what worked for me was this solution: https://stackoverflow.com/a/76515841/22103209

Re: Errors Using Selenium/Chromedriver in DataBricks

Kaizen — Wed, 13 Mar 2024 15:41:23 GMT

this is what im using to install currently

# !/bin/bash

# Script: selenium_init.sh

# Description: Installs Chrome and Chromedriver with path assignment for Selenium.

set -x

sudo apt update

sudo apt upgrade -y 2>upgrade_errors.log

# install require chrome libraries

sudo apt-get update

sudo apt-get install fonts-liberation

sudo apt-get install libgbm1 -y

sudo apt-get install libu2f-udev -y

# create directory

TMP_DIR="tmp"

CHROME_DIR="${TMP_DIR}/chrome"

CHROMEDRIVER_DIR="${TMP_DIR}/chromedriver"

# do a clean install - delete all old drivers

rm -rf ${CHROME_DIR}

rm -rf ${CHROMEDRIVER_DIR}

mkdir -p ${CHROME_DIR}

mkdir -p ${CHROMEDRIVER_DIR}

# install chrome -> ref https://www.baeldung.com/linux/chrome-installation-terminal

cd ${CHROME_DIR}

# alternative google chrome download below is better for supporting latest version

# wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

# sudo dpkg -i google-chrome-stable_current_amd64.deb

# sudo apt-get install -f -y

# sudo apt-get update --fix-missing

# download chrome from same source as chromedriver

wget https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/120.0.6099.109/linux64/chrome-linux64.zip

unzip chrome-linux64.zip

.chrome-linux64/chrome

google-chrome --version || echo "look into google-chrome install"

# install chromedriver -> find all driver files here: https://googlechromelabs.github.io/chrome-for-testing/

cd ..

cd ${CHROMEDRIVER_DIR}

# download chromedriver

curl -SL https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/120.0.6099.109/linux64/chromedriver-linux64.zip -o chromedriver-linux64.zip

unzip chromedriver-linux64.zip

sudo chown root:root /databricks/driver/tmp/chromedriver

sudo chmod +x /databricks/driver/tmp/chromedriver

# TODO: path assignment is not persistant -> Likely permissions issue or databricks compute engine limitation

# Current workaround -> move to preassigned system path

# sudo export PATH=$PATH:/databricks/driver/tmp/chromedriver/chromedriver-linux64/

# copy file to a location defined in path

echo pwd

sudo cp -r chromedriver-linux64/chromedriver /databricks/.pyenv/bin

cd ..

sudo cp -r chrome/chrome-linux64 /databricks/.pyenv/bin

echo "copied file over- new file dir"

cd /databricks/.pyenv/bin

echo "Script execution completed successfully"

echo "Chrome version:" $(google-chrome --version)

echo "Chromedriver version:" $(chromedriver --version)