topic Re: Upload of .bin file >400mb in Data Engineering

Upload of .bin file >400mb

oleh_v — Thu, 05 Sep 2024 10:16:42 GMT

I try to upload to local workspace folder with .bin extension.
It is required to have it locally.

I tried load from DBFS, but loading files over 265mb is not allowed with cluster.
I tried to upload manually but failed with same error "OSError: [Errno5] Input/output error".
also i tried to compress file but since this is binary file compression didn't change significantly file size and manual upload failed as previous attempt.

How can I upload that .bin file to local storage of my workspace?

Re: Upload of .bin file >400mb

szymon_dybczak — Thu, 05 Sep 2024 10:34:37 GMT

Hi @oleh_v ,

You can try below approach, its supports data up to 2GB. For example using curl:

# Parameters databricks_workspace_url="<databricks-workspace-url>" personal_access_token="<personal-access-token>" local_file_path="<local_file_path>" # ex: /Users/foo/Desktop/file_to_upload.png dbfs_file_path="<dbfs_file_path>" # ex: /tmp/file_to_upload.png overwrite_file="<true|false>" curl --location --request POST https://${databricks_workspace_url}/api/2.0/dbfs/put \ --header "Authorization: Bearer ${personal_access_token}" \ --form contents=@${local_file_path} \ --form path=${dbfs_file_path} \ --form overwrite=${overwrite_file}

Upload large files using DBFS API 2.0 and PowerShell - Databricks

Re: Upload of .bin file >400mb

Kartheek_Katta — Fri, 06 Sep 2024 19:44:36 GMT

Hello Slash,

Thank you for your response. I'm encountering the same issue as described. I tried running the provided code in my Databricks workspace, but I received an error. My question is how the script is expected to access local files, especially since the file I’m trying to upload ("pytorch_model.bin") has both read and write permissions on my local machine.

Additionally, once I successfully upload the file to DBFS using the API, can I access and read the file directly from my code, or will I need to use a separate API to retrieve it?

Any guidance or clarification on these points would be greatly appreciated.

# Parameters
databricks_workspace_url="<url>"
personal_access_token="<token>"
local_file_path="C:\\Users\\<Username>\\Downloads\\pytorch_model.bin"
dbfs_file_path="pytorch_model.bin"
overwrite_file="true"