Re: Databricks Workspace import api size limitatio...

pavansharma36 · ‎07-08-2024

As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.

But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit on size of content which is 10mb. When I try from UI to upload lib I can upload with size more than 10mb but not from api.

Not able to find any other way to upload files using api, can anyone share details how to get this done with api.

Thanks

Giri-Patcham · ‎07-08-2024

@pavansharma36 You can try volumes and use Files API to upload
https://docs.databricks.com/api/azure/workspace/files

View solution in original post

pavansharma36 · ‎07-10-2024

PUT <DATABRICKS_HOST>/api/2.0/fs/directories?directory_path=/Workspace/Shared/iw_cgu1prrv/test

gives

{
    "error_code": "BAD_REQUEST",
    "message": "Invalid path:",
    "details": [
        {
            "@type": "type.googleapis.com/google.rpc.ErrorInfo",
            "reason": "FILES_API_INVALID_PATH",
            "domain": "filesystem.databricks.com",
            "metadata": {
                "validation_error": ""
            }
        }
    ]
}

Tried multiple values

eg. /Volumes/samples/default/test

Giri-Patcham · ‎07-10-2024

@pavansharma36 Try something like this

PUT <DATABRICKS_HOST>/api/2.0/fs/directories/Volumes/samples/default/test

pavansharma36 · ‎07-12-2024

Looks like volumes not supported for Standard tier.

{
    "error_code": "BAD_REQUEST",
    "message": "Unity Catalog is not available for feature tier STANDARD_TIER.",
    "details": [
        {
            "@type": "type.googleapis.com/google.rpc.ErrorInfo",
            "reason": "FILES_API_MANAGED_CATALOG_FEATURE_DISABLED",
            "domain": "filesystem.databricks.com",
            "metadata": {
                "unity_catalog_error_message": "Unity Catalog is not available for feature tier STANDARD_TIER."
            }
        }
    ]
}

Not able to create Workspace directories from files api

Whats point of exposing files api onto standard tier when its not usable.

NaraKris_40883 · ‎10-16-2024

Can we use this API to upload files to /workspace/ location? Any sample curl request ? I am using

-X PUT https://<HOST_NAME>/api/2.0/fs/files/Workspace/Shared/jars/all.jar and getting

{

"error_code" : "BAD_REQUEST",

"message" : "Invalid path:",

"details" : [ {

"@type" : "type.googleapis.com/google.rpc.ErrorInfo",

"reason" : "FILES_API_INVALID_PATH",

"domain" : "filesystem.databricks.com",

"metadata" : {

"validation_error" : ""

}

} ]

Databricks Workspace import api size limitation