topic springml sftp with spark 3.x in Data Engineering

springml sftp with spark 3.x

Unimog — Thu, 24 Apr 2025 03:35:49 GMT

Is there a version of springml spark-sftp that works with spark 3.x and scala 2.12? If so can you point me to it or how to load it in my compute?

Re: springml sftp with spark 3.x

Louis_Frolio — Thu, 24 Apr 2025 18:02:07 GMT

The SpringML Spark-SFTP library does not natively support Apache Spark 3.x and Scala 2.12. The library has not been actively maintained, with the last documented commit made in April 2019. This outdated state results in several issues:

Lack of Support for Spark 3.x: It is acknowledged that the SpringML Spark-SFTP library is incompatible with Spark 3.x
Incorrect Implementation: The library is designed as a DataSource rather than as a Hadoop FileSystem, which affects the support for common formats like CSV, Parquet, and JSON in a Spark platform
Unsupported File System Schemes: The library only supports the hdfs:// file system scheme, making it incompatible with the Databricks runtime
No Further Maintenance: The lack of recent updates from the maintainers makes this library an unreliable dependency for modern Spark and Scala environments

Hope this helps, Louis.

Re: springml sftp with spark 3.x

Unimog — Thu, 24 Apr 2025 18:24:02 GMT

Thanks! Any alternatives you recommend for sftp within databricks?

Re: springml sftp with spark 3.x

Louis_Frolio — Thu, 24 Apr 2025 18:40:12 GMT

For Python you might want to look at Paramiko, it seems that it might be an option. You could also look at ETL tools like Airbyte, Rivery, CData, etc.