Can external tables be created backed by current cloud files without ingesting files in Databricks?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-19-2025 01:27 AM
Hi,
We have huge amount of parquet files in s3 with the path pattern <bucket>/<customer>/yyyy/mm/dd/hh/.*.parquet.
The question is can I create a external table in Unity Catalog from this external location without actually ingesting the files? Like what it can be done in AWS Athena:
CREATE EXTERNAL TABLE tbl(
<schema>
)
PARTITION BY (...)
LOCATION
's3://<bucket>/'I have tried a couple of ways to create such a table resulting in errors either as "there is not delta commit logs" or "DELTA_CREATE_TABLE_WITH_NON_EMPTY_LOCATION".
In other words, I don't want to save raw data in Databricks again since we have them in s3. However, I want to be able to query the raw data from Databricks.