cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Workspace import api size limitation

pavansharma36
New Contributor III

As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.

But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit on size of content which is 10mb. When I try from UI to upload lib I can upload with size more than 10mb but not from api.

Not able to find any other way to upload files using api, can anyone share details how to get this done with api.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Giri-Patcham
New Contributor III
New Contributor III
4 REPLIES 4

Giri-Patcham
New Contributor III
New Contributor III

@pavansharma36  You can try volumes and use Files API to upload 
https://docs.databricks.com/api/azure/workspace/files

PUT <DATABRICKS_HOST>/api/2.0/fs/directories?directory_path=/Workspace/Shared/iw_cgu1prrv/test

gives 

 

{
    "error_code": "BAD_REQUEST",
    "message": "Invalid path:",
    "details": [
        {
            "@type": "type.googleapis.com/google.rpc.ErrorInfo",
            "reason": "FILES_API_INVALID_PATH",
            "domain": "filesystem.databricks.com",
            "metadata": {
                "validation_error": ""
            }
        }
    ]
}

 

Tried multiple values

eg. /Volumes/samples/default/test

@pavansharma36 Try something like this 

PUT <DATABRICKS_HOST>/api/2.0/fs/directories/Volumes/samples/default/test

pavansharma36
New Contributor III

Looks like volumes not supported for Standard tier.

{
    "error_code": "BAD_REQUEST",
    "message": "Unity Catalog is not available for feature tier STANDARD_TIER.",
    "details": [
        {
            "@type": "type.googleapis.com/google.rpc.ErrorInfo",
            "reason": "FILES_API_MANAGED_CATALOG_FEATURE_DISABLED",
            "domain": "filesystem.databricks.com",
            "metadata": {
                "unity_catalog_error_message": "Unity Catalog is not available for feature tier STANDARD_TIER."
            }
        }
    ]
}

Not able to create Workspace directories from files api

Whats point of exposing files api onto standard tier when its not usable.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!