topic Creating External Table from partitioned parquet table in Data Engineering

Creating External Table from partitioned parquet table

data_extractor — Wed, 20 Mar 2024 15:42:59 GMT

I am trying to create an external table in catalog using parquet where the parquet file is partitioned

I have tried using the below syntax,

%sql

CREATE TABLE table_name

(

col1 type1,

col2 type2,

col3 type3,

)

USING parquet

PARTITIONED BY ( col4 type4)

LOCATION "s3://<bucket-name>/<folder-name>/parquet_file/"

which gives success in creating the table

but no data is shown when i try to select the table

Can someone let me know what am i doing wrong.

Re: Creating External Table from partitioned parquet table

MichTalebzadeh — Wed, 20 Mar 2024 20:51:16 GMT

This is my guess

Your code (LOCATION "s3://<bucket-name>/<folder-name>/parquet_file/) is likely pointing to a single Parquet file instead of the entire folder containing partitioned data.
try this

CREATE TABLE table_name ( col1 type1, col2 type2, col3 type3, ) USING parquet PARTITIONED BY (col4 type4) LOCATION "s3://<bucket-name>/<folder-name>/" -- Include trailing slash for the folder

Re: Creating External Table from partitioned parquet table

Chizzy — Tue, 24 Sep 2024 09:47:26 GMT

@data_extractor , How were you able to fix this problem? I am having same issue now