Louis_Frolio
Databricks Employee
Databricks Employee

The SpringML Spark-SFTP library does not natively support Apache Spark 3.x and Scala 2.12. The library has not been actively maintained, with the last documented commit made in April 2019. This outdated state results in several issues:

  1. Lack of Support for Spark 3.x: It is acknowledged that the SpringML Spark-SFTP library is incompatible with Spark 3.x

  2. Incorrect Implementation: The library is designed as a DataSource rather than as a Hadoop FileSystem, which affects the support for common formats like CSV, Parquet, and JSON in a Spark platform

  3. Unsupported File System Schemes: The library only supports the hdfs:// file system scheme, making it incompatible with the Databricks runtime

  4. No Further Maintenance: The lack of recent updates from the maintainers makes this library an unreliable dependency for modern Spark and Scala environments

Hope this helps, Louis.

View solution in original post