topic Re: Using Selenium Chrome Driver in Databricks, runs the first time but fails after that in Data Engineering

Using Selenium Chrome Driver in Databricks, runs the first time but fails after that

Tripalink — Mon, 08 Aug 2022 17:56:40 GMT

I have a notebook that uses a Selenium Web Driver for Chrome and it works the first time I run the notebook. If I run the notebook again, it will not work and gives the error message:

WebDriverException: Message: unknown error: unable to discover open pages Stacktrace: #0 0x560b13ce5e89 <unknown>

Here is the code that results in the error message:

chromedriver_path = '/local_disk0/tmp/chromedriver'

chromebinary_path = '/usr/bin/google-chrome'

s = Service(ChromeDriverManager().install())

options = Options()

options.binary_location = chromebinary_path

options.add_argument('--disable-gpu')

options.add_argument('--no-sandbox')

#options.add_argument('--remote-debugging-port=9222')

options.add_argument("--remote-debugging-port=9230")

options.add_argument('--disable-dev-shm-usage')

options.headless = True

driver = webdriver.Chrome(service=s, options=options)

Also, at the end of my notebook I have the following code which may be part of the problem, not sure:

driver.stop_client()

driver.close()

driver.quit()

s.stop()

What can I change to be able to run the notebook as many times as I would like without having it give that error message.

Re: Using Selenium Chrome Driver in Databricks, runs the first time but fails after that

Tripalink — Thu, 11 Aug 2022 00:39:45 GMT

Here is the solution that I discovered:

s = Service(chromedriver_path)

and at the end of the notebook I run:

driver.quit()

and I got rid of driver.stop_client(), driver.close(), and s.stop()

The driver.quit() closes all chrome windows and stops the driver and allows me to run the script again without having to restart the cluster.

Re: Using Selenium Chrome Driver in Databricks, runs the first time but fails after that

Hubert-Dudek — Wed, 09 Nov 2022 14:25:58 GMT

Hi, @Dagart Allison . I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL