topic Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ? in Data Engineering

How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Mon, 20 Jun 2022 08:51:42 GMT

Hi Team,

We are wondering if there is a recommended way to install the chromium browser and chrome driver on Databricks Runtime 10.4 and above ?

I have been through the site and have come across several links to this effect, but they all seem to be installing the Chromium Browser from the official Canonical PPA (ppa:canonical-chromium-builds/stage)

FYI - Please use the contents in link at your own risk -

(https://community.databricks.com/s/question/0D53f00001qEWtxCAG/chromedriver-installation-in-databricks)

From my understanding, the DBX runtime 10.4 is based on Ubuntu 20.04 - focal for which builds do not seem to be available in that PPA. Maybe because Ubuntu moved from official deb to installing Chromium as a snap ?

So questions are as follows

Best way to install Chromium Browser in DBX Runtime 10.4 ?
Should we consider the snap package which seems to be available ? (`apt search chromium-browser`) Will it work well with Selenium and the official Chrome Driver ?
Should we consider any other sources for the Chromium Browser such as (`ppa:phd/chromium-browser`) or the google chrome browser directly ? Will they be safe ? Any license issues ?
Does selenium support any other browsers ? Any other chromium based browsers or firefox with geckodriver ?

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

User16764241763 — Mon, 20 Jun 2022 11:41:02 GMT

Hello @Bharath Kumar Ramachandran

You should be able to use the methods defined here

https://linuxize.com/post/how-to-install-chromium-web-browser-on-ubuntu-20-04/

Section: Installing Chromium with apt

Per this documentation

It supports a number of browsers (Google Chrome 12+, Internet Explorer 7,8,9,10, Safari 5.1+, Opera 11.5, Firefox 3+) and operating systems (Windows, Mac, Linux/Unix).

https://www.browserstack.com/guide/selenium-webdriver-tutorial#:~:text=It%20supports%20a%20number%20of,%2C%20Ruby%2C%20Python%2C%20PHP.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Mon, 20 Jun 2022 12:18:08 GMT

Thank you @Arvind Ravish for your response.

As mentioned in my post, Chromium is no longer available as a deb in the ubuntu archives. The link you had shared is a little old.

When I search in the Databricks Terminal, we receive results showing Chromium Browser as snap and not as a deb. Hence the question.

/databricks/driver# apt search chromium-browser
Sorting... Done
Full Text Search... Done
chromium-browser/focal-updates 1:85.0.4183.83-0ubuntu0.20.04.2 amd64
  Transitional package - chromium-browser -> chromium snap
 
chromium-browser-l10n/focal-updates 1:85.0.4183.83-0ubuntu0.20.04.2 all
  Transitional package - chromium-browser-l10n -> chromium snap

Would be really nice if Databricks can provide a recommended way to handle this, since this is a very common use case and there are so many such questions around this topic in our community forums itself...

Also regarding the other browsers, since we are using databricks, requirement is limited to linux and open source browsers only. I hear that not many websites are tested with firefox, hence a little hesitant there...

Thank you...

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Tue, 28 Jun 2022 06:43:33 GMT

Hi @Kaniz Fatma , Thank you so much for your response.

I have already been through the link you had shared.

It seems to be installing google-chrome directly from the Google Servers.

But I have been told that google-chrome is proprietary and chromium-browser is open source. Usage of proprietary software in code might cause license issues or some other issues such as telemetry etc. right ? Hence I was looking for some way we could use the open source version i.e. Chromium on which chrome is based on.

Is there a databricks hosted repository available for such use cases or does selenium work with the snap version of chromium-browser ? I hope I am not the only DBX user using DBX runtime 10.4 and trying to use chromium-browser...

Update:

Have tried installing `chromium-browser` and `chromium-chromedriver` through init scripts, but they are not working since the snap processes are not available. Directly trying `snap install chromium` is also not working...

Thank you for all the help.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Prabakar — Tue, 05 Jul 2022 14:02:46 GMT

Hi @Bharath Kumar Ramachandran please refer to askUbuntu here. The Chromium browser is only available as a snap in 19.10 and above. If you don't want to install the Chromium snap package, see How to install Chromium without snap?

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Wed, 06 Jul 2022 06:47:57 GMT

Hi @Prabakar Ammeappin ,

Thank you very much for your response.

yes you are correct, Chromium became a snap in 19.10. Databricks runtimes 9.1 and 10.4 both have the Ubuntu version of 20.04.

I am ok with using the snap version of Chromium, but it is not installing properly...

When I try to install using the below command in DBX runtime 10.4 through init script...

apt update

apt install chromium-browser chromium-chromedriver

The following additional packages will be installed:

apparmor liblzo2-2 snapd squashfs-tools udev

Suggested packages:

apparmor-profiles-extra apparmor-utils zenity | kdialog

The following NEW packages will be installed:

apparmor chromium-browser chromium-chromedriver liblzo2-2 snapd

squashfs-tools udev

0 upgraded, 7 newly installed, 0 to remove and 24 not upgraded.

Need to get 36.3 MB of archives

The init script is completing, but we have log that says as follows.

=> Installing the chromium snap

==> Checking connectivity with the snap store

===> System doesn't have a working snapd, skipping

Also Chromium is not working, when I search for it using which chromium or which chromium-browser, it does not show up.

Is there any other way to get the snap installed and working first before installing ?

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Wed, 06 Jul 2022 06:52:18 GMT

Hi @Prabakar Ammeappin ,

I have also gone through the other options of installing Chromium without a snap. But we are finding the options either too complex or not from an official source. I found the following options from the link you had shared. Which option would Databricks recommend ?

Option 01 - Install from Debian Repo
1. Is it safe to mix packages from Debian and Ubuntu - Messing source lists seems risky ? Even if we pin items, is it safe enough ?
Option 02 - ppa:saiarcot895/chromium-beta - Unofficial PPA - Is it safe ? Can we trust it to remain patched and up to date ?
Option 03 - Google Chrome - Possible License Issues considering that Chrome is not open source ?
Other Options - Flatpak, Nix Installs, Linux Mint Install.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Prabakar — Wed, 06 Jul 2022 11:43:28 GMT

Hi @Bharath Kumar Ramachandran considering the options I would prefer either to go with Option 1 or 4.

Option 2 is unofficial and I won't consider going for it. It's always better to use the official one as we can trust the provider.

Option 3 should be good but as there is a doubt with the licensing then I would not take a risk. If you want to go with this option then do all possible research w.r.t the license part and then go for it.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Wed, 06 Jul 2022 13:39:46 GMT

Hi @Prabakar Ammeappin

Thank you for your response.

We will try options 01 and 04, however can you please let us know why the snap install is not working. We are using DBX runtime 10.4.

apt update

apt install chromium-browser chromium-chromedriver

snapd is not working as expected.

Please see my response from earlier.

It would be preferable and safe to use the options available in the Ubuntu repos by default instead of bringing in additional repos from outside.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Will1 — Thu, 07 Jul 2022 10:16:56 GMT

Chromium can be installed using the Software application and via the command line.

Installing Chromium using Software (GUI)

Click on the Software tool in Fedora.
Search for Chromium Web Browser.
Click on Install.

Regards,

Willjoe

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Prabakar — Sat, 16 Jul 2022 00:23:31 GMT

Hi @Bharath Kumar Ramachandran i need to test this and shall get back to you with my findings or a workaround.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Mon, 18 Jul 2022 06:50:48 GMT

Thank you @Prabakar Ammeappin

Using the Ubuntu sources would be the best case scenario since we can be sure about the authenticity of the source.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Thu, 28 Jul 2022 10:24:12 GMT

Hi @Prabakar Ammeappin ,

Just wanted to check if you got a chance to test the Chromium Snap Install issue ?

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Vidula — Thu, 25 Aug 2022 08:54:55 GMT

Hi there @Bharath Kumar Ramachandran

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Thu, 25 Aug 2022 09:50:38 GMT

Hi @Vidula Khanna, We still have not identified a way to install Open Source Chromium Browser in DBX runtime 10.4.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Prabakar — Thu, 25 Aug 2022 14:47:15 GMT

Hi @Bharath Kumar Ramachandran @Vidula Khanna , I tried all possible ways to install chromium but at one or the other place it is failing on something. Going back and troubleshooting it is really a time-consuming task. To make it simple, you can try using the docker container services. Install chromium at the container level and use the image.

https://hub.docker.com/layers/standard/databricksruntime/standard/10.4-LTS/images/sha256-caee5e0d586d874f06da010ab330a750210d4896c1103474f332d318c01b6b79?context=explore

https://docs.databricks.com/clusters/custom-containers.html

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Tue, 30 Aug 2022 06:48:04 GMT

Thank you so much for your response and apologies for all the trouble.

We had considered the custom runtime options suggested, but it got turned down because the effort to create and maintain the custom runtime was not something we could do at this moment. Also maintenance fixes that flow down time to time are also something to consider. For now, we have planned to either pause or move the requirement to a different/existing option.

Thank you once again for all the help.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

Hubert-Dudek — Wed, 09 Nov 2022 14:27:25 GMT

Hi, @Bharath Kumar Ramachandran I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

swrd — Fri, 11 Nov 2022 01:45:23 GMT

Has anyone figured out how to get Selenium to work on Databricks seamlessly?

Beginning to question whether it's a common occurrence for the Databricks support team to not get back to disgruntled users on time...

This request has been opened for some months with no satisfactory response.

Does anyone know any online resource that could guide us to this pertinent issue? Any response at this stage would be appreciated.

Re: How to install Chromium Browser and Chrome Driver on DBX runtime 10.4 and above ?

ranged_coop — Tue, 15 Nov 2022 10:10:10 GMT

Thank you @Hubert Dudek for your script.

If possible, suggest including a reference to the below link and description just in case Google changes some of the steps so that in the future if the process changes, users would still be able to figure it out.

https://www.chromium.org/getting-involved/download-chromium/

Not-as-easy steps:

Head to https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html
Choose your platform: Mac, Win, Linux, ChromiumOS
Pick the Chromium build number you'd like to use
The latest one is mentioned in the
LAST_CHANGE
1. file
Download the zip file containing Chromium
There is a binary executable within to run