Databricks Community

himanmon · 07-16-2024

Hello. I have a file over 100GB. Sometimes this is on the cluster's local path, and sometimes it's on the volume.And I want to send this to another path on the volume, or to the s3 bucket. dbutils.fs.cp('file:///tmp/test.txt', '/Volumes/catalog/schem...

himanmon · 07-14-2024

I'm not sure if this is the right place to post my question. If not, please let me know where I should post my question. I want to download large files from the web from Databricks' master(driver) node. For example, I fetch a file over 150GB via API ...

himanmon · 07-04-2024

I'm trying to run a workflow with job compute.Job compute needs to be pending for about 5 to 7 minutes before executing the workflow. I think it takes time to find a suitable instance in the cloud, configure the environment, install libraries, etc.An...

himanmon · 07-17-2024

Hi @szymon_dybczak , Thank you for your answer.Of course, I restarted the cluster. However, it still uses 10mb blocks.I understand that compression could be an option, but there doesn't seem to be any solution when the compressed file exceeds 100GB.

himanmon · 07-14-2024

I didn't know that instances with more CPU cores and memory size also had larger local storage. I know from your answer that more expensive instances have more storage, but it's a burden because it increases the cost.However, I absolutely need to dow...

himanmon · 07-08-2024

Thank you for your answer. That means pending time is also included in the cost, right?

Databricks Community

User Stats

User Activity

Can I move a single file larger than 100GB using dbtuils fs?

How can I increase the hard capacity of the master node?

When does the cost of JOB COMPUTE start to be calculated?

Re: Can I move a single file larger than 100GB using dbtuils fs?

Re: How can I increase the hard capacity of the master node?

Re: When does the cost of JOB COMPUTE start to be calculated?