topic Re: How to access hard disk attached to cluster? in Data Engineering

How to access hard disk attached to cluster?

alvaro_databric — Mon, 13 Feb 2023 14:39:29 GMT

Hi,

I am using the VM family Lasv3, which incorporate a NVMe SSD. I would like to take advantage of this huge amount of space but I cannot find where this disk is mounted. Does someone know where this disk is mounted and if it can be used as local driver disk is used (to use commands such as os.listdir, uncompressing zips, etc?

Thanks in advance!

Re: How to access hard disk attached to cluster?

Anonymous — Sat, 11 Mar 2023 03:13:09 GMT

Hi @Alvaro Moure

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to access hard disk attached to cluster?

JosiahJohnston — Fri, 01 Nov 2024 23:00:19 GMT

Great question; I've been trying to hunt that down also. `/local_disk0` looks like a good candidate, but it has restricted access and I can't confirm or use.

Would love to learn a solution someday. This is a big need for hybrid workflows & libraries common in my organization. Spark is great for some things, but we also use other frameworks extensively.