Multiple volumes from same external location?

dsmoore — Tue, 03 Dec 2024 16:38:47 GMT

Hey all,

Do you know if it's possible to create multiple volumes referencing the same s3 bucket from the same external location?

For example, if I have two workspaces (test and prod) testing different versions of pipeline code but with static data I'd like to be able to read from the same external volume. After setting it up in one workspace, I go to create the volume in the other and get the following error:

Input path url '<s3 path>' overlaps with other external tables or volumes within 'CreateVolume' call. Conflicting tables/volumes: <original_volume>

Any guidance or is this just a limitation with volumes in Databricks?

Thanks!

Re: Multiple volumes from same external location?

ozaaditya — Wed, 04 Dec 2024 05:49:20 GMT

Yes, it is a limitation, and it is not possible to create multiple volumes referencing the same S3 bucket. This restriction ensures consistency and prevents conflicts when accessing the same data source.

Possible Solution:

Use subdirectories within the same bucket. For example:
- s3://bucket/prod
- s3://bucket/test

Below are some links for reference:
https://community.databricks.com/t5/data-governance/external-locations/td-p/67356

https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error

topic Multiple volumes from same external location? in Administration & Architecture

Multiple volumes from same external location?

Re: Multiple volumes from same external location?

Possible Solution: