Using Selenium Chrome Driver in Databricks, runs the first time but fails after that

[Tripalink]

I have a notebook that uses a Selenium Web Driver for Chrome and it works the first time I run the notebook. If I run the notebook again, it will not work and gives the error message:

WebDriverException: Message: unknown error: unable to discover open pages Stacktrace: #0 0x560b13ce5e89 <unknown>

Here is the code that results in the error message:

chromedriver_path = '/local_disk0/tmp/chromedriver'

chromebinary_path = '/usr/bin/google-chrome'

s = Service(ChromeDriverManager().install())

options = Options()

options.binary_location = chromebinary_path

options.add_argument('--disable-gpu')

options.add_argument('--no-sandbox')

#options.add_argument('--remote-debugging-port=9222')

options.add_argument("--remote-debugging-port=9230")

options.add_argument('--disable-dev-shm-usage')

options.headless = True

driver = webdriver.Chrome(service=s, options=options)

Also, at the end of my notebook I have the following code which may be part of the problem, not sure:

driver.stop_client()

driver.close()

driver.quit()

s.stop()

What can I change to be able to run the notebook as many times as I would like without having it give that error message.

[Tripalink]

Here is the solution that I discovered:

s = Service(chromedriver_path)

and at the end of the notebook I run:

driver.quit()

and I got rid of driver.stop_client(), driver.close(), and s.stop()

The driver.quit() closes all chrome windows and stops the driver and allows me to run the script again without having to restart the cluster.

[Hubert-Dudek]

Hi, @Dagart Allison​ . I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL