topic Re: Jobs indefinitely pending with libraries install in Data Engineering

Jobs indefinitely pending with libraries install

martindlarsson — Tue, 20 Aug 2024 15:09:52 GMT

I think I found a bug where you get Pending indefinitely on jobs that has a library requirement and the user of the job does not have Manage permission on the cluster.

In my case I was trying to start a dbt job with dbt-databricks=1.8.5 as library. The cluster did not have the library installed and I had only "Can restart" access to the cluster. When I started the job it was Pending in about 20 minutes until I canceled it and the result in the Runs view Status: Internal error and Error Code: LibraryInstallationError.

Once I gave myself the "Can Manage" on the cluster the job started and the Library successfully installed.

Re: Jobs indefinitely pending with libraries install

VZLA — Fri, 27 Dec 2024 18:30:46 GMT

Thanks for your feedback!

Just checking is this still an issue for you? would you share more details? if I wanted to reproduce this for example.

Re: Jobs indefinitely pending with libraries install

martindlarsson — Wed, 08 Jan 2025 14:54:56 GMT

This is not an issue, just wanted to let you know and and if other users has jobs pending forever. I think you have all the information in my description to reproduce.