Well I am looking at

ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()

If the library did not succeed in installing, it cannot compile.

You mention beam being included in the jar, but are you sure it is a complete jar (the forum does not allow met to use the word f-a-t :-)?

Also can you check this:

https://kb.databricks.com/en_US/jobs/job-fails-no-library