My organization has an S3 bucket mounted to the databricks filesystem under

/dbfs/mnt

. When using Databricks runtime 5.5 and below, the following logging code works correctly:

log_file = '/dbfs/mnt/path/to/my/bucket/test.log' logger = logging.getLogger('test-logger') logger.setLevel(logging.INFO) handler = logging.FileHandler(str(log_file)) handler.setLevel(logging.INFO) logger.addHandler(handler)

logger.info('test')

After upgrading to Databricks runtime 6.1, the above code produces a logging error "OSError: [Errno 95] Operation not supported". Here's the stack trace that is printed:

Traceback (most recent call last):
  File "/databricks/python/lib/python3.7/logging/__init__.py", line 1038, in emit
    self.flush()
  File "/databricks/python/lib/python3.7/logging/__init__.py", line 1018, in flush
    self.stream.flush()
OSError: [Errno 95] Operation not supported

The strange thing is that regular Python file I/O works fine with the same file. (i.e. I can

open()

and

write()

to that filepath successfully.) Any idea what's going on?