This is the nature of shared clusters . You can install libraries for a task but isolation is not guaranteed . if a library is already installed on the cluster it will take priority over what defined for the task.

Any reason you cant use job clusters . They are cheap and provide the isolation needed in your case . Another options is using Serverless jobs with environment specs .