Yes, thank you! It is really nice explanation.

But then I must return to my initial question - how to guarantee that the library will be installed on this ephemeral job cluster.

The solution to which I came now is to use Databricks Container Services

and run job cluster using my custom image with preinstalled library.