topic Re: Databricks Pipeline SDK misisng fields in Data Engineering

Databricks Pipeline SDK misisng fields

aranjan99 — Thu, 25 Sep 2025 23:27:35 GMT

Looking at the Databricks Java SDK for Pipeline events, I see that the Rest API returns a details field that has the same information as event log details. But this is not surfaced in SDK, should be a small change to add it. Is that something which can be prioritized?

Same for the start/end time fields in Pipeline Updates API, its present in Rest API but missing in SDK, is this something that can be added

Re: Databricks Pipeline SDK misisng fields

ManojkMohan — Fri, 26 Sep 2025 12:51:43 GMT

The start and end time fields in the Pipeline Updates API are currently present in the Databricks REST API but are not yet supported (i.e., not included or mapped) in the Databricks Java SDK as of September 2025.

This means:

You can see these fields (start_time and end_time) when you directly call the REST API.

However, these fields are missing from the Java SDK's API response models, so you cannot access them through the SDK currently.

The SDK maintainers have not officially added support for these fields in the SDK release versions to date. An interesting work around to look at - https://community.databricks.com/t5/data-engineering/how-to-get-pipeline-update-duration-programmatically/td-p/132885

Re: Databricks Pipeline SDK misisng fields

aranjan99 — Fri, 26 Sep 2025 18:03:01 GMT

As I understand, Databricks is the maintainer of the SDK? If so is this something that can be added soon?
I looked at the workaround but that doesnt work for me as our client has only given us API access and cannot enable logs

Re: Databricks Pipeline SDK misisng fields

ManojkMohan — Mon, 29 Sep 2025 12:39:31 GMT

Yes, Databricks is the official maintainer of the Java SDK, which means enhancements and field mappings are ultimately prioritized by their product and engineering teams. Currently, the details field for pipeline event logs and the start/end time fields for Pipeline Updates are both returned by the REST API, but have not yet been exposed in the Java SDK’s response models or data classes as of September 2025. This is a known feature parity gap between the REST API and SDKs, commonly discussed in the community. https://community.databricks.com/t5/data-engineering/how-to-get-pipeline-update-duration-programmatically/td-p/132885

If so is this something that can be added soon?

@Advika @Sujitha May be something for the product team to consider ...

Re: Databricks Pipeline SDK misisng fields

Advika — Wed, 01 Oct 2025 13:40:30 GMT

Thanks for tagging, @ManojkMohan!

@aranjan99, could you please raise this as a feature request here?
I’ve also flagged this internally and will keep you updated.