topic Re: Why the SFTP ingest doesn't work? in Data Engineering

Why the SFTP ingest doesn't work?

eyalo — Sun, 14 May 2023 12:06:19 GMT

Hi, I did the following code but it seems like the cluster is running for a long period of time and then stops without any results.

Attached my following code: (I used 'com.springml.spark.sftp' library and install it as Maven)

Also i whitelisted my local machine's IP on the SFTP server. Pherhaps there is another whitlisting to do in case the databricks engine running on different IP?

Appriciate any help here, Even an alternative solution.

Thanks.

Re: Why the SFTP ingest doesn't work?

eyalo — Wed, 17 May 2023 07:33:20 GMT

Hi @Debayan Mukherjee , thanks for your reply.

After restarting my cluster i got the following error: 'java.lang.NoClassDefFoundError: scala/Product$class'

As i understand, It seeme like it doesn't recognize my class even-though i installed it to my cluster's libraries:

com.springml:spark-sftp_2.10:1.0.2

com.springml:spark-sftp_2.11:1.1.3

Do you know what could be the reason?

FYI - I did it based on this documentation: https://github.com/springml/spark-sftp

Re: Why the SFTP ingest doesn't work?

eyalo — Mon, 22 May 2023 11:03:30 GMT

@Debayan Mukherjee Hi, I don't know if you got my reply so i am bouncing my message to you again.

Thanks.

Re: Why the SFTP ingest doesn't work?

eyalo — Tue, 30 May 2023 07:00:21 GMT

Hi @Debayan Mukherjee , Certainly. After i checked it seems like the library com.springml:spark-sftp_2.11:1.1.3 doesn't support my cluster's runtime version (11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) but i coun't upgdate newer version for this library so perhaps that's why it didn't work.

Either way, I will raise a ticket. thanks!

Re: Why the SFTP ingest doesn't work?

Debayan — Mon, 15 May 2023 16:29:26 GMT

Hi, Could you please update the error here?

Please tag @Debayan with your next comment so that I will get notified. Thank you!

Re: Why the SFTP ingest doesn't work?

Debayan — Tue, 30 May 2023 06:14:00 GMT

Hi, Sorry for the delay! I think the issue can surface due to various reasons. We can investigate the whole cluster and triage on this. Could you please raise a supportcase with us so that we can continue the investigation?

Thanks again!

Re: Why the SFTP ingest doesn't work?

Debayan — Mon, 05 Jun 2023 07:01:13 GMT

@Eyal Ozeri . DBR version can be the issue here. Thanks!