I think I found a bug where you get Pending indefinitely on jobs that has a library requirement and the user of the job does not have Manage permission on the cluster.
In my case I was trying to start a dbt job with dbt-databricks=1.8.5 as library. The cluster did not have the library installed and I had only "Can restart" access to the cluster. When I started the job it was Pending in about 20 minutes until I canceled it and the result in the Runs view Status: Internal error and Error Code: LibraryInstallationError.
Once I gave myself the "Can Manage" on the cluster the job started and the Library successfully installed.
