topic Re: Databricks Workspace import api size limitation in Data Engineering

Databricks Workspace import api size limitation

pavansharma36 — Mon, 08 Jul 2024 14:15:38 GMT

As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.

But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit on size of content which is 10mb. When I try from UI to upload lib I can upload with size more than 10mb but not from api.

Not able to find any other way to upload files using api, can anyone share details how to get this done with api.

Thanks

Re: Databricks Workspace import api size limitation

Giri-Patcham — Mon, 08 Jul 2024 15:08:55 GMT

@pavansharma36 You can try volumes and use Files API to upload
https://docs.databricks.com/api/azure/workspace/files

Re: Databricks Workspace import api size limitation

pavansharma36 — Wed, 10 Jul 2024 10:08:30 GMT

PUT <DATABRICKS_HOST>/api/2.0/fs/directories?directory_path=/Workspace/Shared/iw_cgu1prrv/test

gives

{ "error_code": "BAD_REQUEST", "message": "Invalid path:", "details": [ { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "FILES_API_INVALID_PATH", "domain": "filesystem.databricks.com", "metadata": { "validation_error": "" } } ] }

Tried multiple values

eg. /Volumes/samples/default/test

Re: Databricks Workspace import api size limitation

Giri-Patcham — Wed, 10 Jul 2024 14:59:32 GMT

@pavansharma36 Try something like this

PUT <DATABRICKS_HOST>/api/2.0/fs/directories/Volumes/samples/default/test

Re: Databricks Workspace import api size limitation

pavansharma36 — Fri, 12 Jul 2024 07:35:53 GMT

Looks like volumes not supported for Standard tier.

{ "error_code": "BAD_REQUEST", "message": "Unity Catalog is not available for feature tier STANDARD_TIER.", "details": [ { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "FILES_API_MANAGED_CATALOG_FEATURE_DISABLED", "domain": "filesystem.databricks.com", "metadata": { "unity_catalog_error_message": "Unity Catalog is not available for feature tier STANDARD_TIER." } } ] }

Not able to create Workspace directories from files api

Whats point of exposing files api onto standard tier when its not usable.

Re: Databricks Workspace import api size limitation

NaraKris_40883 — Thu, 17 Oct 2024 05:03:02 GMT

Can we use this API to upload files to /workspace/ location? Any sample curl request ? I am using

-X PUT https://<HOST_NAME>/api/2.0/fs/files/Workspace/Shared/jars/all.jar and getting

{

"error_code" : "BAD_REQUEST",

"message" : "Invalid path:",

"details" : [ {

"@type" : "type.googleapis.com/google.rpc.ErrorInfo",

"reason" : "FILES_API_INVALID_PATH",

"domain" : "filesystem.databricks.com",

"metadata" : {

"validation_error" : ""

}

} ]