05-14-2023 05:06 AM
Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results.
Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)
Also i whitelisted my local machine's IP on the SFTP server. Pherhaps there is another whitlisting to do in case the databricks engine running on different IP?
Appriciate any help here, Even an alternative solution.
Thanks.
05-15-2023 09:29 AM
Hi, Could you please update the error here?
Please tag @Debayan with your next comment so that I will get notified. Thank you!
05-17-2023 12:33 AM
Hi @Debayan Mukherjee , thanks for your reply.
After restarting my cluster i got the following error: 'java.lang.NoClassDefFoundError: scala/Product$class'
As i understand, It seeme like it doesn't recognize my class even-though i installed it to my cluster's libraries:
com.springml:spark-sftp_2.10:1.0.2
com.springml:spark-sftp_2.11:1.1.3
Do you know what could be the reason?
FYI - I did it based on this documentation: https://github.com/springml/spark-sftp
05-22-2023 04:03 AM
@Debayan Mukherjee Hi, I don't know if you got my reply so i am bouncing my message to you again.
Thanks.
05-29-2023 11:14 PM
Hi, Sorry for the delay! I think the issue can surface due to various reasons. We can investigate the whole cluster and triage on this. Could you please raise a supportcase with us so that we can continue the investigation?
Thanks again!
05-30-2023 12:00 AM
Hi @Debayan Mukherjee , Certainly. After i checked it seems like the library com.springml:spark-sftp_2.11:1.1.3 doesn't support my cluster's runtime version (11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) but i coun't upgdate newer version for this library so perhaps that's why it didn't work.
Either way, I will raise a ticket. thanks!
06-05-2023 12:01 AM
@Eyal Ozeri . DBR version can be the issue here. Thanks!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group