topic Re: INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error with S3 paths in Administration & Architecture

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error with S3 paths

shawnbarrick — Tue, 25 Jul 2023 12:23:08 GMT

We're trying to read from an S3 bucket using unity catalog and are selectively getting "INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error" errors within the same bucket.

This works:

"dbutils.fs.ls("s3://BUCKETNAME/dev/health")"

But within the same bucket we get the location overlap error when running: "dbutils.fs.ls("s3://BUCKETNAME/dev/claims/")"

I reviewed the article https://kb.databricks.com/en_US/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error, but it's unclear to me why one path works. Is there something I should check other than IAM permissions for the bucket in question?

Re: INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error with S3 paths

Debayan — Wed, 26 Jul 2023 07:32:08 GMT

Hi,
Could you please elaborate on the issue here?
Running the list command on a managed directory is not supported in Unity Catalog. Catalog/schema storage locations are reserved for managed storage.
Please tag @Debayan with your next comment which will notify me. Thanks!

Re: INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error with S3 paths

shawnbarrick — Wed, 26 Jul 2023 12:35:04 GMT

Thanks for the response. We are trying to to verify S3 buckets access, as well as helping a user troubleshooting a permissions issues.

I run this command I get a list of the bucket contents for that folder: "dbutils.fs.ls("s3://BUCKETNAME/dev/health")"

Likewise this will show the beginning of a file:

"dbutils.fs.head("s3://gradientai-databricks/dev/health/Q2_2023/1_30.csv")"

However one of our users gets "PermissionError: Forbidden" when running this code:

"import pandas as pd"
"import pickle"
"data = pd.read_pickle('s3://BUCKETNAME/dev/health/Q2_2023/1_30.csv')"
"display(data)"