Spark Declarative Pipelines use in All-purpose compute?

[ChristianRRL]

Hi there, I know Spark Declarative Pipelines (previously DLT) has undergone some changes since last year and is even now open source (announcement). For a long while, I know that SDP/DLT was locked to only working with job compute. I'm wondering, with recent changes, will SDP now (or in the near future) work with All-purpose compute?

[MoJaMa]

Not today and likely not in future as well.

1. If you create a pipeline yourself, the pipeline creates it's own compute. This can be classic (ie you choose the nodes/node-types) or serverless (recommended)

2. If you define a Streaming Table or Materialized View in DBSQL, it create a serverless DLT/LSDP pipeline behind the scenes.

3. There is plan to support it from Serverless Interactive (but not classic All Purpose) during the development lifecycle.

The main reason is that it needs a custom runtime (forked from DBR with various configs tuned/set) and so your "normal" DBR that you use for classic all-purpose(or jobs) will not be sufficient.

[ChristianRRL]

Since Spark Declarative Pipelines are now open source, is it conceivable that a team may be able to leverage the open-source version to have a working version of SDP that works with All-purpose compute (assuming a future DBR LTS that has 4.1.x and above)? Is there something that I'm not seeing that may impede the open-source version from working in All-purpose compute?