Paramiko SFTP Get fails on databricks file system

Kayla
Valued Contributor II

I have an SFTP server I need to routinely download Excel files from and put into GCP cloud storage buckets.
Every variation of the filepath to either my GCP path or just the dbfs in-built file system is giving an error of " [Errno 2] No such file or directory:" Has anyone else encountered this before?

 

 

import paramiko

client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(host, port=port, username=username, password=password)
sftp = client.open_sftp()

remote_path = "/Upload/filename.txt"
local_path = "/dbfs/filename.txt"

 

 

 

Kayla
Valued Contributor II

I've discovered the cause of this issue. The path of '/dbfs/{mount-name}/filename.file_extension' is functional, but what I'm actually running into is a restriction due to the cluster set to "Shared" access mode:

  • Cannot use R, RDD APIs, or clients that directly read the data from cloud storage, such as DBUtils.

Changing the cluster mode resolved the issue.

View solution in original post

jose_gonzalez
Databricks Employee
Databricks Employee

Thank you for sharing the solution. Many more users will find this information very useful.