Comment
12-12-2024
01:11 PM
12-12-2024
01:11 PM
This looks exactly like what I've been hoping for! However, the tutorial as shown here doesn't seem to work for me.
- The repo https://s01.oss.sonatype.org/content/repositories/iounitycatalog-1016/ does not exist and 404s. Luckily, unitycatalog-spark is published elsewhere on sonatype
- There seems to be a version mismatch in the dependent hadoop libraries.
- Spark 3.5.{0-3} are all compiled against hadoop 3.3.4.
- unitycatalog-spark versions 0.2.{0,1} depend on spark 3.5.3.
- unitycatalog-spark also depend on hadoop-client-runtime 3.4.0
- hadoop libraries 3.4.0 have an updated FSBuilder that has an `optLong` method that is not present in 3.3.4. This results in `java.lang.NoSuchMethodError` if this tutorial is attempted with spark 3.3.4 and `java.lang.NumberFormatException` if upgraded to 3.4.0.
Is this tutorial possible or was unitycatalog-spark released with incompatible dependencies? Thanks!