Jackson version conflict

[Shimon]

Hi,

I am trying to implement the Spark TableProvider api and i am experiencing a jar conflict (I am using the 17.3 runtime).

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.15.2 requires Jackson Databind version >= 2.15.0 and < 2.16.0 - Found jackson-databind version 2.18.2
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
	at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:908)
	at com.fasterxml.jackson.databind.ObjectMapper.registerModules(ObjectMapper.java:1110)
	at com.fasterxml.jackson.databind.ObjectMapper.findAndRegisterModules(ObjectMapper.java:1194)

In the release notes it shows that the "jackson-databind" library should be 2.18.2,
It seems like the shading in the `shaded.databricks.org.apache.hudi.com` was not done well.
Trying to understand the bug i run the following command

%sh find /databricks/jars -name "*hudi*.jar" -exec jar tf {} \; | grep "META-INF/services/com.fasterxml.jackson"

which returned 'META-INF/services/com.fasterxml.jackson.databind.Module'. this helps me conclude that the bug in a shading issue in the databricks hudi package.

Could someone take a look at this?
Thank you.  

[Abeshek]

This is a well-documented issue, and it highlights how dependency conflicts in managed runtimes can quickly become a blocker for teams extending Spark beyond standard use cases.

We see similar challenges when organizations build custom providers or integrate open-source frameworks on Databricks, especially as runtimes evolve. In many cases, the complexity comes less from the feature itself and more from managing runtime behavior and hidden dependency interactions at scale.

Curious how you’re planning to move forward here and whether this is being handled via Databricks support or internally. We’ve helped teams navigate similar runtime-level challenges, so happy to share what has worked if helpful.

[Shimon]

For now we are trying to contact Databricks, 
In worst case scenario we were planning to shade the dependencies we need.

would love to hear what has worked for you.

Best,
Shimon