Unexpected ProtoBuf Version Changes on DBR 15.4 LTS Causing databricks-feature-client Failures

[guilhermeneves]

Hi all,

I noticed an issue starting from Friday, Aug 15 through Monday, Aug 18. Our clusters running DBR 15.4 LTS experienced unexpected upgrades of the protobuf library:

• Friday: protobuf was at version 4.24.1 (as documented here: DBR 15.4 LTS release notes)
• Saturday: it was upgraded to 5.29.5
• Sunday: it moved again to 6.32.0
• Monday morning: it reverted back to 4.24.1

This behavior is not documented in the maintenance updates.

The change broke our usage of databricks-feature-client with errors like:

File /Workspace/Repos/.internal/.../utils.py:12 from databricks.feature_engineering import FeatureEngineeringClient ... ImportError: cannot import name 'service' from 'google.protobuf'

The stack trace points to duplicate_proto_detection() inside databricks.feature_engineering, which fails due to incompatibility between databricks-feature-client and the unexpected protobuf version upgrades.

After Monday morning, things stabilized again with protobuf 4.24.1, and the issue disappeared.

Question: Were there unannounced runtime changes or hotfixes over the weekend that explain these Cluster DBR 15.4LTS protobuf version fluctuations? If so, could they be documented, since they directly impact feature engineering workloads?

Thanks,
Guilherme

[Advika · Accepted solution]

Hello @guilhermeneves!

Could you confirm if pip was used to install any Python packages in your workload?
The version shifts you’re seeing are likely related to unpinned pip installs: when package versions aren’t pinned, dependency changes from third-party libraries can impact workload behavior. Pinning dependencies is recommended to prevent this.

[guilhermeneves]

Thanks, Advika! your answer helped a lot.

The issue turned out to be caused by a dbt-databricks minor update. Although we had pinned the package to avoid major upgrades, the minor update still introduced the problem.

So yes, libraries was added to the cluster policy to install the dependencies, but the unexpected behavior came specifically from the minor version change.

Thank you,

Guilherme