Hello,

One solution I have found is to use job cluster Policies. With that, you are able to design groups of job clusters, where libraries can be installed just as in all-purpose computes. 

In my case, I needed these job clusters to run through Azure Data Factory and all I had to do once I defined my policies, was to add the policy ID to my Databricks linked service.

Hope this helps,

Sacha