Add Raw Tabless to Unity Catalog?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2024 02:48 PM
Hello everyone,
Hope someone here can help me out as I am a bit stuck 🙂
I am currently loading some data from number of sources to parquet using ADF. This ends up in a ADLS2 storage account as .parquet. This is the staging / landing zone area.
From there I am using a databricks notebook to move it from staging to bronze while converting to delta.
What I am trying to do is add those tables using the autoloader so that I can "get it into the catalog" and refer to it in dbt when working from bronze, to silver to gold. I can browse fine, but the moment I try to preview there is an issue.
The error I get is:
Your request failed with status FAILED: [BAD_REQUEST] Reading from a Delta table is not supported with this syntax. If you would like to consume data from Delta, please refer to the docs: read a Delta table (https://docs.microsoft.com/azure/databricks/delta/tutorial#read), or read a Delta table as a stream source (https://docs.microsoft.com/azure/databricks/structured-streaming/delta-lake#table-streaming-reads-an...). The streaming source from Delta is already optimized for incremental consumption of data.
And after that I can't get any further because I can't preview the table to add it.
Any input or thoughts would be great and thanks for any responses!
