topic Re: Legacy Pipeline? in Data Engineering

Legacy Pipeline?

aonurdemir — Mon, 24 Feb 2025 07:55:32 GMT

What is this legacy pipeline warning? I cannot find anything related on the internet and related forums. Thanks in advance.

Re: Legacy Pipeline?

Alberto_Umana — Mon, 24 Feb 2025 16:03:28 GMT

Hi @aonurdemir,

Thanks for your question. Are you using run_as configuration for pipeline, and how did you provision it, via UI or DABs?

Re: Legacy Pipeline?

aonurdemir — Tue, 25 Feb 2025 07:06:21 GMT

Hi @Alberto_Umana,

I provision it via UI. And, this is the json config of it:

{ "id": "...", "pipeline_type": "WORKSPACE", "development": true, "continuous": false, "channel": "PREVIEW", "libraries": [ ... ], "name": "Prod Lookup Pipeline", "serverless": true, "catalog": "...", "configuration": { "helpers-path": "...", "s3-base-topic-path-prefix": "...", "s3-base-topic-path-daily-suffix": "..." }, "target": "prod" }

Re: Legacy Pipeline?

aonurdemir — Tue, 25 Feb 2025 08:28:33 GMT

I realized that creating pipeline button has changed. Databricks has brought new options to the create pipeline button. After selecting create etl option, the new json configuration is seen as the following:

{ "development": true, "continuous": false, "channel": "CURRENT", "photon": true, "libraries": [ ... ], "name": "Test pipe", "serverless": true, "catalog": "...", "schema": "dev", "data_sampling": false, "budget_policy_id": "..." }

The diffs are those:

- "photon": true,
- "data_sampling": false,
- "budget_policy_id": "..."

Is this relevant?

Re: Legacy Pipeline?

aonurdemir — Tue, 25 Feb 2025 08:41:59 GMT

Ah I've found the issue. There is one more diff:

This is the new key-value for target schema:

"schema": "dev",

This is the old key-value for target schema:

    "target": "prod"

After changing the key from "target" to "schema", the warning has gone. Thanks anyway.