cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Why the SFTP ingest doesn't work?

eyalo
New Contributor II

Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results.

Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)

Also i whitelisted my local machine's IP on the SFTP server. Pherhaps there is another whitlisting to do in case the databricks engine running on different IP?

Appriciate any help here, Even an alternative solution.

Thanks.

image 

6 REPLIES 6

Debayan
Databricks Employee
Databricks Employee

Hi, Could you please update the error here?

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

eyalo
New Contributor II

Hi @Debayan Mukherjee​ , thanks for your reply.

After restarting my cluster i got the following error: 'java.lang.NoClassDefFoundError: scala/Product$class'

As i understand, It seeme like it doesn't recognize my class even-though i installed it to my cluster's libraries:

com.springml:spark-sftp_2.10:1.0.2

com.springml:spark-sftp_2.11:1.1.3

image.png 

Do you know what could be the reason?

FYI - I did it based on this documentation: https://github.com/springml/spark-sftp

eyalo
New Contributor II

@Debayan Mukherjee​ Hi, I don't know if you got my reply so i am bouncing my message to you again.

Thanks.

Debayan
Databricks Employee
Databricks Employee

Hi, Sorry for the delay! I think the issue can surface due to various reasons. We can investigate the whole cluster and triage on this. Could you please raise a supportcase with us so that we can continue the investigation?

Thanks again!

eyalo
New Contributor II

Hi @Debayan Mukherjee​ , Certainly. After i checked it seems like the library com.springml:spark-sftp_2.11:1.1.3 doesn't support my cluster's runtime version (11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) but i coun't upgdate newer version for this library so perhaps that's why it didn't work.

Either way, I will raise a ticket. thanks!

Debayan
Databricks Employee
Databricks Employee

@Eyal Ozeri​ . DBR version can be the issue here. Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group