topic Re: ChromeDriver installation in Databricks in Data Engineering

ChromeDriver installation in Databricks

AJ270990 — Fri, 08 Apr 2022 06:01:05 GMT

I am working on a Webscraping logic and need to install Chrome driver. How can I install it in the Databricks workbook ?

Re: ChromeDriver installation in Databricks

Hubert-Dudek — Mon, 11 Apr 2022 20:47:56 GMT

@Abhishek Jain , In the clusters library tab, please install PyPi chromedriver-binary==83.0 (or higher, probably version in the script can also be updated)

To cluster init script, please add /databricks/scripts/selenium-install.sh, which you create using the below code.

Then, in the databricks notebook code, please use something similar to the second code below.

dbutils.fs.mkdirs("dbfs:/databricks/scripts/")
dbutils.fs.put("/databricks/scripts/selenium-install.sh","""
#!/bin/bash
apt-get update
apt-get install chromium-browser=91.0.4472.101-0ubuntu0.18.04.1 --yes
wget https://chromedriver.storage.googleapis.com/91.0.4472.101/chromedriver_linux64.zip -O /tmp/chromedriver.zip
mkdir /tmp/chromedriver
unzip /tmp/chromedriver.zip -d /tmp/chromedriver/
""", True)
display(dbutils.fs.ls("dbfs:/databricks/scripts/"))

from selenium import webdriver
chrome_driver = '/tmp/chromedriver/chromedriver'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-dev-shm-usage') 
chrome_options.add_argument('--homedir=/dbfs/tmp')
chrome_options.add_argument('--user-data-dir=/dbfs/selenium')
# prefs = {"download.default_directory":"/dbfs/tmp",
#          "download.prompt_for_download":False
# }
# chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)

Re: ChromeDriver installation in Databricks

Buga — Tue, 21 Jun 2022 00:02:39 GMT

Hi,

I'm trying to use this solution to use seleium in databricks, but i can't.

Can you help me ?

Follow image

Re: ChromeDriver installation in Databricks

Hubert-Dudek — Wed, 09 Nov 2022 14:27:45 GMT

Hi, @Gustavo Queiroz I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL

Re: ChromeDriver installation in Databricks

Evan_MCK — Wed, 21 Jun 2023 15:29:54 GMT

What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. You can see all the details here: https://stackoverflow.com/questions/69192050/using-selenium-within-databricks-chrome-not-reachable/76515841#76515841