- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2024 06:01 PM
I'm not sure if this is the right place to post my question. If not, please let me know where I should post my question.
I want to download large files from the web from Databricks' master(driver) node. For example, I fetch a file over 150GB via API request.
Of course I know this isn't a beautiful thing, but for some reason I have to do it.
The problem is that the master node's disk capacity is insufficient. I tried adding an EBS volume in the cluster settings, but it doesn't seem to be the option I want.
I am trying to download a 150 GB file to the /tmp path on the master node's local file system. I would like to know how to increase insufficient disk space.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2024 07:18 PM
Hi @himanmon,
If you 100% sure that you can't download this file to storage account configured with unity catalog and you want it directly on driver node local storage, then why can't you just increase local disk space by choosing a larger instance type?Select an instance type with more local storage capacity for the master node and you should be fine.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2024 07:54 PM
I didn't know that instances with more CPU cores and memory size also had larger local storage.
I know from your answer that more expensive instances have more storage, but it's a burden because it increases the cost.
However, I absolutely need to download the file to local storage for some reason, so it's pretty sad if there's no other way.
(The reason I mentioned earlier is, I need to increment data line by line in append mode in a txt file using some API.)
Thank you for your answer.
