cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks Pipeline SDK misisng fields

aranjan99
Contributor

Looking at the Databricks Java SDK for Pipeline events, I see that the Rest API returns a details field that has the same information as event log details. But this is not surfaced in SDK, should be a small change to add it. Is that something which can be prioritized?

Same for the start/end time fields in Pipeline Updates API, its present in Rest API but missing in SDK, is this something that can be added

4 REPLIES 4

ManojkMohan
Honored Contributor

The start and end time fields in the Pipeline Updates API are currently present in the Databricks REST API but are not yet supported (i.e., not included or mapped) in the Databricks Java SDK as of September 2025.

This means:

You can see these fields (start_time and end_time) when you directly call the REST API.

However, these fields are missing from the Java SDK's API response models, so you cannot access them through the SDK currently.

The SDK maintainers have not officially added support for these fields in the SDK release versions to date.   An interesting work around to look at - https://community.databricks.com/t5/data-engineering/how-to-get-pipeline-update-duration-programmati...

As I understand, Databricks is the maintainer of the SDK? If so is this something that can be added soon?
I looked at the workaround but that doesnt work for me as our client has only given us API access and cannot enable logs

Yes, Databricks is the official maintainer of the Java SDK, which means enhancements and field mappings are ultimately prioritized by their product and engineering teams. Currently, theโ€ฏdetails field for pipeline event logs and the start/end time fields for Pipeline Updates are both returned by the REST API, but have not yet been exposed in the Java SDKโ€™s response models or data classes as of September 2025. This is a known feature parity gap between the REST API and SDKs, commonly discussed in the community.  https://community.databricks.com/t5/data-engineering/how-to-get-pipeline-update-duration-programmati...

 If so is this something that can be added soon?

@Advika  @Sujitha  May be something for the product team to consider ...

Advika
Databricks Employee
Databricks Employee

Thanks for tagging, @ManojkMohan!

@aranjan99, could you please raise this as a feature request here?
Iโ€™ve also flagged this internally and will keep you updated.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now