- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2023 11:52 AM
I have an SFTP server I need to routinely download Excel files from and put into GCP cloud storage buckets.
Every variation of the filepath to either my GCP path or just the dbfs in-built file system is giving an error of " [Errno 2] No such file or directory:" Has anyone else encountered this before?
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(host, port=port, username=username, password=password)
sftp = client.open_sftp()
remote_path = "/Upload/filename.txt"
local_path = "/dbfs/filename.txt"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-06-2023 05:22 AM
I've discovered the cause of this issue. The path of '/dbfs/{mount-name}/filename.file_extension' is functional, but what I'm actually running into is a restriction due to the cluster set to "Shared" access mode:
- Cannot use R, RDD APIs, or clients that directly read the data from cloud storage, such as DBUtils.
Changing the cluster mode resolved the issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-12-2023 03:36 PM
Thank you for sharing the solution. Many more users will find this information very useful.