Comment
New Contributor

This looks exactly like what I've been hoping for! However, the tutorial as shown here doesn't seem to work for me.

  1. The repo https://s01.oss.sonatype.org/content/repositories/iounitycatalog-1016/ does not exist and 404s. Luckily, unitycatalog-spark is published elsewhere on sonatype
  2. There seems to be a version mismatch in the dependent hadoop libraries.
    • Spark 3.5.{0-3} are all compiled against hadoop 3.3.4.
    • unitycatalog-spark versions 0.2.{0,1} depend on spark 3.5.3.
    • unitycatalog-spark also depend on hadoop-client-runtime 3.4.0
    • hadoop libraries 3.4.0 have an updated FSBuilder that has an `optLong` method that is not present in 3.3.4. This results in `java.lang.NoSuchMethodError` if this tutorial is attempted with spark 3.3.4 and `java.lang.NumberFormatException` if upgraded to 3.4.0.

Is this tutorial possible or was unitycatalog-spark released with incompatible dependencies? Thanks!