Jackson version conflict

Shimon
New Contributor II

‎12-14-2025 04:37 AM

Hi,

I am trying to implement the Spark TableProvider api and i am experiencing a jar conflict (I am using the 17.3 runtime).

 

com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.15.2 requires Jackson Databind version >= 2.15.0 and < 2.16.0 - Found jackson-databind version 2.18.2
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
	at shaded.databricks.org.apache.hudi.com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
	at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:908)
	at com.fasterxml.jackson.databind.ObjectMapper.registerModules(ObjectMapper.java:1110)
	at com.fasterxml.jackson.databind.ObjectMapper.findAndRegisterModules(ObjectMapper.java:1194)

 

In the release notes it shows that the "jackson-databind" library should be 2.18.2,
It seems like the shading in the `shaded.databricks.org.apache.hudi.com` was not done well.
Trying to understand the bug i run the following command

 

 

%sh find /databricks/jars -name "*hudi*.jar" -exec jar tf {} \; | grep "META-INF/services/com.fasterxml.jackson"

 

 

which returned 'META-INF/services/com.fasterxml.jackson.databind.Module'. this helps me conclude that the bug in a shading issue in the databricks hudi package.

Could someone take a look at this?
Thank you.  

Labels:
0 Kudos

Abeshek
Databricks Partner

‎12-15-2025 05:20 AM

This is a well-documented issue, and it highlights how dependency conflicts in managed runtimes can quickly become a blocker for teams extending Spark beyond standard use cases.

We see similar challenges when organizations build custom providers or integrate open-source frameworks on Databricks, especially as runtimes evolve. In many cases, the complexity comes less from the feature itself and more from managing runtime behavior and hidden dependency interactions at scale.

Curious how you’re planning to move forward here and whether this is being handled via Databricks support or internally. We’ve helped teams navigate similar runtime-level challenges, so happy to share what has worked if helpful.

Regards,
Abeshek CH
abeshek.cherukuri@kanerika.com
0 Kudos

Shimon
New Contributor II

‎12-15-2025 11:11 PM

For now we are trying to contact Databricks, 
In worst case scenario we were planning to shade the dependencies we need.

would love to hear what has worked for you.

Best,
Shimon

0 Kudos

emanuele_m
Databricks Employee
Databricks Employee

‎02-25-2026 04:05 AM

Hi,

this problem occurs if you have dynamic module registration, e.g.

new ObjectMapper().findAndRegisterModules()

and the way to solve it is to use something like this instead

val jsonMapper = new ObjectMapper()
jsonMapper.registerModule(DefaultScalaModule)
0 Kudos