DLT piplines with UC
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 08:47 AM
I try to run a (very simple) DLT pipeline in with a resulting materialized table is published in UC schema with a managed storage location defined (within an existing EXTERNAL LOCATION). Accoding to the documentation:
Publishing to schemas that specify a managed storage location is supported only in the preview channel.
So - with no surprises - running this pipeline in the current channel ends with:
com.databricks.pipelines.common.CustomException: [DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE]
Schemas with specified storage locations are not currently supported for UC enabled pipelines.
But when I switch to the preview channel, it does not work either, but this time the error is different:
com.databricks.pipelines.common.CustomException: [DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE]
Schema tstexternal has a specified storage location that does not match the pipeline's root location: None
Is there a chance for it to make it working? Can I setup "the pipeline's root location" somehow?
Any help welcome.
Thx
Pawel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 11:01 AM - edited 03-21-2024 11:01 AM
@ws4100e Did you select the target Catalog and Schema from the pipeline settings?
For persisting on UC managed schemas, currently we need to specify schema and select catalog.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-22-2024 02:35 AM
Yes, this is exactly what I did - I selected a catalog and a schema (with a storage location assigned) within it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-22-2024 06:05 AM
I was receiving the same error however it was resolved after selecting right schema and right permissions while creating fresh pipeline.
Can you please share the code writing the table?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-22-2024 07:43 AM
I try this for the simplest possible pipeline - I use Autoloader to read CSV files from Azure storage. I've got sufficient privileges on selected schema.
CREATE OR REFRESH STREAMING LIVE TABLE employees_raw_el
AS SELECT *
FROM cloud_files(
"abfss://container1@sa12345678.dfs.core.windows.net/csvs/",
"csv",
map(
"header", "true",
"delimiter", ";",
"inferSchema", "true"
)
);
CREATE OR REFRESH LIVE TABLE employees_bronze_el
COMMENT "Test external location"
TBLPROPERTIES ("table.usage" = "tests")
AS SELECT *
FROM live.employees_raw_el;
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-12-2024 01:19 AM
Did this get resolved? I am getting the same issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-12-2024 05:16 AM
Unfortunately not :(.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2024 04:45 AM
I was facing the same issue and was able to solve the problem by selecting channel as 'preview' instead of 'current'
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-22-2024 04:17 AM
As I mentioned above - I use preview channel, so it must be something else 😕
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2024 03:49 AM