Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-29-2024 05:47 AM
I am currently using this library in my databricks where i am facing challenge to load the data from sftp
local_path = f'/Workspace/Users/<user.email>/Scard_etl/SFTP_LOCAL/JDL/'
sftp_df = spark.read.format("com.springml.spark.sftp")\
.option("host", jdl_sftp.host)\
.option("username", jdl_sftp.user)\
.option("password", jdl_sftp.password)\
.option("fileType", "txt")\
.load("/dsdata/jdl/files/jdl_settlements.txt")
sftp_df.write.mode('overwrite').text(local_path)
I am getting error below
Py4JJavaError: An error occurred while calling o504.load. : java.lang.NoClassDefFoundError: scala/Product$class at com.springml.spark.sftp.DatasetRelation.<init>(DatasetRelation.scala:24)
As per my research do I need to upgrade the version of library if yes where can find the upgraded version of this as i need to attach it to my cluster in databricks
I am getting error below
Py4JJavaError: An error occurred while calling o504.load. : java.lang.NoClassDefFoundError: scala/Product$class at com.springml.spark.sftp.DatasetRelation.<init>(DatasetRelation.scala:24)
As per my research do I need to upgrade the version of library if yes where can find the upgraded version of this as i need to attach it to my cluster in databricks
com.springml:spark-sftp_2.11:1.0.3
