topic Re: Paramiko SFTP Get fails on databricks file system in Data Engineering

Paramiko SFTP Get fails on databricks file system

Kayla — Tue, 26 Sep 2023 18:52:44 GMT

I have an SFTP server I need to routinely download Excel files from and put into GCP cloud storage buckets.
Every variation of the filepath to either my GCP path or just the dbfs in-built file system is giving an error of " [Errno 2] No such file or directory:" Has anyone else encountered this before?

import paramiko client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(host, port=port, username=username, password=password) sftp = client.open_sftp() remote_path = "/Upload/filename.txt" local_path = "/dbfs/filename.txt"

Re: Paramiko SFTP Get fails on databricks file system

Kayla — Fri, 06 Oct 2023 12:22:20 GMT

I've discovered the cause of this issue. The path of '/dbfs/{mount-name}/filename.file_extension' is functional, but what I'm actually running into is a restriction due to the cluster set to "Shared" access mode:

Cannot use R, RDD APIs, or clients that directly read the data from cloud storage, such as DBUtils.

Changing the cluster mode resolved the issue.

Re: Paramiko SFTP Get fails on databricks file system

jose_gonzalez — Thu, 12 Oct 2023 22:36:21 GMT

Thank you for sharing the solution. Many more users will find this information very useful.