Community Custom Connector - Defining Non-Serverless Job Compute Runtime?

ChristianRRL
Honored Contributor

yesterday

Hi there,

I'm exploring Community Custom Connectors and I'm wondering about the Compute. By default, it's set to Serverless:

ChristianRRL_0-1784905334067.png

But due to IT network limitations at our company, we cannot use serverless. 

In the Compute settings, I can untoggle the Serverless option, and it reveals what I assume must be Job Compute settings:

ChristianRRL_1-1784905449577.png

This is good, but I'm wondering if there's a way to explicitly define the "Runtime Version" on this Job Compute?

0 Kudos

balajij8
Esteemed Contributor

yesterday

Hi ChristianRRL,

Because Community Custom Connectors are backed by Lakeflow pipelines, the compute layer has specific architectural constraints. The underlying pipeline execution framework runs on a custom, managed version of DBR that Databricks continually updates. Properties like DBR aren't generally exposed or user-settable because the platform needs to guarantee compatibility with the ingestion framework, seamlessly apply security patches and ensure consistency across pipeline updates.

Instead of traditional DBR versions, Lakeflow pipelines operate on pipeline environment versions. If you were looking to pin a runtime specifically to control the Python version or pre-installed libraries, you can check if the available environment versions meet your requirements. You can also set the Product Edition (Core/Pro/Advanced) based on your pipeline type.

More details here

View solution in original post

adnan_alvee
Databricks Employee
Databricks Employee

yesterday - last edited yesterday

You generally cannot select DBR version in Lakeflow / Spark Declarative Pipelines because the pipeline runtime is Databricks-managed and versionless. Databricks auto-upgrades the runtime for pipelines.

The settings has a JSON view, and some advanced options are only available in JSON.  And the supported control you get is the channel field (current or preview), not an arbitrary spark_version like on Jobs clusters like below:

adnan_alvee_0-1784908979543.png

current = the normal production channel. It uses the current pipeline runtime version, is the default, and Databricks recommends it for production workloads.

preview = the next/upcoming runtime channel. It’s meant for testing your pipeline against upcoming runtime changes before they roll into current.


As of the current release notes, the channels map roughly to:
current → DBR 17.3
preview → DBR 18




0 Kudos