topic Re: can I use volume for external table location? in Data Engineering

can I use volume for external table location?

PaulineX — Tue, 20 Feb 2024 21:21:49 GMT

Hello,

I have a parquet file test.parquet in the volume volume_ext_test. Tried to create an external table as below, it failed and says it "is not a valid URI".

create table catalog_managed.schema_test.tbl_vol as

select * from parquet.`/Volumes/catalog_managed/schema_test/volume_ext_test/test.parquet`

but I have no problem to create a managed table as below:

create table catalog_managed.schema_test.tbl_vol as

select * from parquet.`/Volumes/catalog_managed/schema_test/volume_ext_test/test.parquet`

Am I doing something wrong? Or it is the limitation?

Thanks.

Re: can I use volume for external table location?

PaulineX — Tue, 20 Feb 2024 21:29:22 GMT

sorry pasted the wrong statement. the external table statement is :

CREATE TABLE px_catalog_managed.schema_test.test_tbl2

USING parquet

OPTIONS (

path '/Volumes/catalog_managed/schema_test/volume_ext_test/test.parquet',

inferSchema 'true'

)

which gives me error

Re: can I use volume for external table location?

AmanSehgal — Tue, 20 Feb 2024 22:03:15 GMT

Hi @PaulineX

As per the documentation, you cannot use volume for storing table data. It's for loading, storing and accessing files.

You cannot use volumes as a location for tables. Volumes are intended for path-based data access only. Use tables for storing tabular data with Unity Catalog.

It's bit ironic here, as delta table data is stored in files as well.

Anyway, as per the documentation, answer to your question is no.

Re: can I use volume for external table location?

PaulineX — Tue, 20 Feb 2024 22:30:29 GMT

@AmanSehgal Thank you for the reply. This is exactly what I want to confirm!

Too bad, I hoped to avoid full path in notebooks.