If anyone else is still interested in this topic, I tried running it with Databricks runtime 18.1 ML. We still have the same memory issue, but we now also get the nice error message for in approach 3 as 

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 14.58 GiB of which 5.62 MiB is free

One thing I didn't discuss in this thread is to try to use the hosted models somehow. If your end goal is a vector index, you can now also bypass this step and simply select embedding model instead of precomputing it. This raises the question on how to reason about host. The hosted models are limited compared to the world wide web and I think those are pay per token instead of runtime.