How to manipulate files in an external location?

Tjomme
Databricks Partner

According to the documentation, the usage of external locations is preferred over the use of mount points.

Unfortunately the basic funtionality to manipulate files seems to be missing.

This is my scenario:

  • create a download folder in an external location if it does not exist:
dbutils.fs.mkdirs(NewPath) does not work --> Operation failed: "This request is not authorized to perform this operation."
  • use API to download zip files from a source and write it to a mounted location using: 
f = open(fullFileName, 'w+b') --> FileNotFoundError: [Errno 2] No such file or directory
f.write(ZipBinaryData)
f.close()
  • loop all zip files to: dbutils.fs.ls does not work: needs to be replaced with LIST
    • unzip them into an extract folder containing JSON files (not tested yet, but using zipfile.ZipFile(fullZipFileName) )
    • load the JSON files into a (raw) managed table (should not be an issue)
    • further process the managed table (should not be an issue)
    • empty extract folder using
dbutils.fs.rm(NewPath,True) --> Operation failed: "This request is not authorized to perform this operation."
  • move zip file to archive folder using
dbutils.fs.mv(NewPath,ArchivePathTrue) --> Operation failed: "This request is not authorized to perform this operation."

Any help or insights on how to get this working with external locations is greatly appreciated!