Databricks Java SDK retrieving job task values

[anabel0]

Greetings,

I have a Job that consists of notebook tasks running python code.

Some of the task set task values using 

dbutils.jobs.taskValues.set(key=key, value=value)

as described here 

How do I retrieve those task values using Databricks Java SDK v0.69.0?

So far I've tried using jobs().getRun(...) with includeResolvedValues=true but it only returns job/base parameters, and the rest of the fields are null.

Run run = workspaceClient.jobs().getRun(
                    new GetRunRequest()
                            .setRunId(taskRunIn)
                            .setIncludeResolvedValues(true));
run.getTasks().forEach(trt -> trt.getResolvedValues().getNotebookTask().getBaseParameters()
                            .forEach((k,v) -> System.out.printf("%s: %s = %s\n", rt.getTaskKey(), k, v)));

Any suggestions would be highly appreciated. 

[stbjelcevic]

Unfortunately you can’t read dbutils.jobs.taskValues directly via the Java SDK. But you could return a JSON payload via dbutils.notebook.exit and read it with getRunOutput. Databricks exposes the notebook “exit” result through Jobs GetRunOutput, then parse the JSON in your Java client.

Python (in the notebook task):

import json
# Set your task values as usual
dbutils.jobs.taskValues.set(key="record_count", value=42)  # example :llmCitationRef[11]

# Emit a JSON payload you want to read externally
dbutils.notebook.exit(json.dumps({
    "record_count": 42,
    "order_status": "Delivered"
}))

Java (client side):

RunOutput out = workspaceClient.jobs().getRunOutput(
    new GetRunOutputRequest().setRunId(taskRunId));
String result = out.getNotebookOutput().getResult();  // JSON string
Map<String, Object> vals = new ObjectMapper()
    .readValue(result, new TypeReference<Map<String, Object>>() {});
System.out.println(vals.get("record_count"));

[anabel0]

Thank you for your reply.

Though dbutils.notebook.exit + getRunOutput approach might be helpful in similar cases, it doesn't suit our case, as not only it requires changing the job code, but also won't work for already existing runs.

I wonder why task values were left out of API/SDK. That data is definitely available, you can see it in the side panel of a run page, fetched with graphql. Any idea if it's in the roadmap, or if there are fundamental reasons why it wasn't included?