Look into Playwrite instead of Selenium. I went through the same process y'all went through here (ended up writing a init script to install the drivers etc)

This is all done for you in playwright. Refer to this post - I hope it helps!!
https://community.databricks.com/t5/community-discussions/using-python-rpa-library-on-databricks/td-...

Oh - and you also no longer have to manage your driver versions 😉 thats a plus!