topic Re: Jackson version conflict in Data Engineering

Jackson version conflict

Shimon — Sun, 14 Dec 2025 12:37:00 GMT

Hi,

I am trying to implement the Spark TableProvider api and i am experiencing a jar conflict (I am using the 17.3 runtime).

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.15.2 requires Jackson Databind version >= 2.15.0 and < 2.16.0 - Found jackson-databind version 2.18.2 at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61) at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46) at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:908) at com.fasterxml.jackson.databind.ObjectMapper.registerModules(ObjectMapper.java:1110) at com.fasterxml.jackson.databind.ObjectMapper.findAndRegisterModules(ObjectMapper.java:1194)

In the release notes it shows that the "jackson-databind" library should be 2.18.2,
It seems like the shading in the `shaded.databricks.org.apache.hudi.com` was not done well.
Trying to understand the bug i run the following command

%sh find /databricks/jars -name "*hudi*.jar" -exec jar tf {} \; | grep "META-INF/services/com.fasterxml.jackson"

which returned 'META-INF/services/com.fasterxml.jackson.databind.Module'. this helps me conclude that the bug in a shading issue in the databricks hudi package.

Could someone take a look at this?
Thank you.

Re: Jackson version conflict

Abeshek — Mon, 15 Dec 2025 13:20:33 GMT

This is a well-documented issue, and it highlights how dependency conflicts in managed runtimes can quickly become a blocker for teams extending Spark beyond standard use cases.

We see similar challenges when organizations build custom providers or integrate open-source frameworks on Databricks, especially as runtimes evolve. In many cases, the complexity comes less from the feature itself and more from managing runtime behavior and hidden dependency interactions at scale.

Curious how you’re planning to move forward here and whether this is being handled via Databricks support or internally. We’ve helped teams navigate similar runtime-level challenges, so happy to share what has worked if helpful.

Re: Jackson version conflict

Shimon — Tue, 16 Dec 2025 07:11:56 GMT

For now we are trying to contact Databricks,
In worst case scenario we were planning to shade the dependencies we need.

would love to hear what has worked for you.

Best,
Shimon

Re: Jackson version conflict

emanuele_m — Wed, 25 Feb 2026 12:05:06 GMT

Hi,

this problem occurs if you have dynamic module registration, e.g.

new ObjectMapper().findAndRegisterModules()

and the way to solve it is to use something like this instead

val jsonMapper = new ObjectMapper() jsonMapper.registerModule(DefaultScalaModule)