Why the SFTP ingest doesn't work?

[eyalo]

Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results.

Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)

Also i whitelisted my local machine's IP on the SFTP server. Pherhaps there is another whitlisting to do in case the databricks engine running on different IP?

Appriciate any help here, Even an alternative solution.

Thanks.

 

[Debayan]

Hi, Could you please update the error here?

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

[eyalo]

Hi @Debayan Mukherjee​ , thanks for your reply.

After restarting my cluster i got the following error: 'java.lang.NoClassDefFoundError: scala/Product$class'

As i understand, It seeme like it doesn't recognize my class even-though i installed it to my cluster's libraries:

com.springml:spark-sftp_2.10:1.0.2

com.springml:spark-sftp_2.11:1.1.3

 

Do you know what could be the reason?

FYI - I did it based on this documentation: https://github.com/springml/spark-sftp

[eyalo]

@Debayan Mukherjee​ Hi, I don't know if you got my reply so i am bouncing my message to you again.

Thanks.

[Debayan]

Hi, Sorry for the delay! I think the issue can surface due to various reasons. We can investigate the whole cluster and triage on this. Could you please raise a supportcase with us so that we can continue the investigation?

Thanks again!

[eyalo]

Hi @Debayan Mukherjee​ , Certainly. After i checked it seems like the library com.springml:spark-sftp_2.11:1.1.3 doesn't support my cluster's runtime version (11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) but i coun't upgdate newer version for this library so perhaps that's why it didn't work.

Either way, I will raise a ticket. thanks!

[Debayan]

@Eyal Ozeri​ . DBR version can be the issue here. Thanks!