topic webscraping with Firefox driver failing - Python code in Get Started Discussions

webscraping with Firefox driver failing - Python code

375721 — Thu, 17 Aug 2023 13:55:56 GMT

Getting below exception while running web scraping script, stuck at installing firefox from snap. Getting apparmor permission issue and tried to reinstall it and still getting the same. please check the error message section.

Dependencies:

Python library: selenium -> added to the cluster

Other dependencies:

Gecko driver installation:

%sh wget https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux-aarch64.tar.gz

%sh tar -xvzf geckodriver-v0.33.0-linux-aarch64.tar.gz

%sh mv geckodriver /tmp

Firefox installation:

%sh /usr/bin/yes | sudo apt update --fix-missing

sudo apt-get --yes --force-yes install firefox

trying to run below sample code in notebook

from selenium import webdriver

from selenium.webdriver.firefox.options import Options

opts = Options()

opts.headless=True

browser = webdriver.Firefox(options=opts, executable_path='/tmp/geckodriver')

browser.get('https://mft.rrc.texas.gov/link/20ff2205-6579-450f-a2ee-cbd37986b557')

error message:
WebDriverException: Message: Process unexpectedly closed with status 1

I did some more research and found the below messages from geckodriver.log

Command '/usr/bin/firefox' requires the firefox snap to be installed. Please install it with: snap install firefox

And when I try to run snap install firefox getting below error.

error: system does not fully support snapd: AppArmor detected but insufficient permissions to use it

Re: webscraping with Firefox driver failing - Python code

-werners- — Thu, 17 Aug 2023 15:17:21 GMT

have you tried using apt instead of snap?

sudo apt install firefox

Re: webscraping with Firefox driver failing - Python code

375721 — Thu, 17 Aug 2023 15:39:19 GMT

if you see my messages, I did install firefox with sudo apt-get --yes --force-yes install firefox .

and then ran the python code block and getting the error as posted in the comment

browser = webdriver.Firefox(options=opts, executable_path='/tmp/geckodriver')

Re: webscraping with Firefox driver failing - Python code

Debayan — Mon, 21 Aug 2023 06:16:09 GMT

Hi,

This is an issue with the dependencies of different packages.

Please try as below:

apt get update && apt get upgrade;

apt install firefox;

Please tag @Debayan with your next comment which will notify me.

Re: webscraping with Firefox driver failing - Python code

-werners- — Mon, 21 Aug 2023 06:45:05 GMT

Sorry I missed that.
You could try to install snap by adjusting/disabling apparmor. But I doubt that will fix anything as the snap package for firefox will be the same as the firefox apt install (except with added dependencies).
Have you tried if firefox is actually working? (/path/to/firefox -headless)
Here is an interesting article: https://hacks.mozilla.org/2017/12/using-headless-mode-in-firefox/