topic DLT piplines with UC in Data Engineering

DLT piplines with UC

ws4100e — Thu, 21 Mar 2024 15:47:56 GMT

I try to run a (very simple) DLT pipeline in with a resulting materialized table is published in UC schema with a managed storage location defined (within an existing EXTERNAL LOCATION). Accoding to the documentation:

Publishing to schemas that specify a managed storage location is supported only in the preview channel.

So - with no surprises - running this pipeline in the current channel ends with:

com.databricks.pipelines.common.CustomException: [DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE] 
Schemas with specified storage locations are not currently supported for UC enabled pipelines.

But when I switch to the preview channel, it does not work either, but this time the error is different:

com.databricks.pipelines.common.CustomException: [DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE] 
Schema tstexternal has a specified storage location that does not match the pipeline's root location: None

Is there a chance for it to make it working? Can I setup "the pipeline's root location" somehow?

Any help welcome.

Thx

Pawel

Re: DLT piplines with UC

data-engineer-d — Thu, 21 Mar 2024 18:01:27 GMT

@ws4100e Did you select the target Catalog and Schema from the pipeline settings?
For persisting on UC managed schemas, currently we need to specify schema and select catalog.

Re: DLT piplines with UC

ws4100e — Fri, 22 Mar 2024 09:35:51 GMT

Yes, this is exactly what I did - I selected a catalog and a schema (with a storage location assigned) within it.

Re: DLT piplines with UC

data-engineer-d — Fri, 22 Mar 2024 13:05:23 GMT

I was receiving the same error however it was resolved after selecting right schema and right permissions while creating fresh pipeline.

Can you please share the code writing the table?

Re: DLT piplines with UC

ws4100e — Fri, 22 Mar 2024 14:43:05 GMT

I try this for the simplest possible pipeline - I use Autoloader to read CSV files from Azure storage. I've got sufficient privileges on selected schema.

CREATE OR REFRESH STREAMING LIVE TABLE employees_raw_el
AS SELECT *
FROM cloud_files(
  "abfss://container1@sa12345678.dfs.core.windows.net/csvs/",
  "csv",
  map(
    "header", "true",     
    "delimiter", ";",
    "inferSchema", "true"
  )
);

CREATE OR REFRESH LIVE TABLE employees_bronze_el
COMMENT "Test external location"
TBLPROPERTIES ("table.usage" = "tests")
AS SELECT *
  FROM live.employees_raw_el;

Re: DLT piplines with UC

DataGeek_JT — Fri, 12 Apr 2024 08:19:21 GMT

Did this get resolved? I am getting the same issue.

Re: DLT piplines with UC

ws4100e — Fri, 12 Apr 2024 12:16:36 GMT

Unfortunately not :(.

Re: DLT piplines with UC

_Architect_ — Wed, 01 May 2024 11:45:31 GMT

I was facing the same issue and was able to solve the problem by selecting channel as 'preview' instead of 'current'

Re: DLT piplines with UC

ws4100e — Wed, 22 May 2024 11:17:02 GMT

As I mentioned above - I use preview channel, so it must be something else 😕

Re: DLT piplines with UC

ImranA — Wed, 06 Nov 2024 11:49:22 GMT

Difference between the Preview and Current Channel?