Databricks Community

eyalo · ‎05-14-2023

Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results.

Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)

Also i whitelisted my local machine's IP on the SFTP server. Pherhaps there is another whitlisting to do in case the databricks engine running on different IP?

Appriciate any help here, Even an alternative solution.

Thanks.

Debayan · ‎05-15-2023

Hi, Could you please update the error here?

Please tag @Debayan with your next comment so that I will get notified. Thank you!

eyalo · ‎05-17-2023

Hi @Debayan Mukherjee , thanks for your reply.

After restarting my cluster i got the following error: 'java.lang.NoClassDefFoundError: scala/Product$class'

As i understand, It seeme like it doesn't recognize my class even-though i installed it to my cluster's libraries:

com.springml:spark-sftp_2.10:1.0.2

com.springml:spark-sftp_2.11:1.1.3

Do you know what could be the reason?

FYI - I did it based on this documentation: https://github.com/springml/spark-sftp

eyalo · ‎05-22-2023

@Debayan Mukherjee Hi, I don't know if you got my reply so i am bouncing my message to you again.

Thanks.

Debayan · ‎05-29-2023

Hi, Sorry for the delay! I think the issue can surface due to various reasons. We can investigate the whole cluster and triage on this. Could you please raise a supportcase with us so that we can continue the investigation?

Thanks again!

eyalo · ‎05-30-2023

Hi @Debayan Mukherjee , Certainly. After i checked it seems like the library com.springml:spark-sftp_2.11:1.1.3 doesn't support my cluster's runtime version (11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) but i coun't upgdate newer version for this library so perhaps that's why it didn't work.

Either way, I will raise a ticket. thanks!