Python mocking dbutils in unittests

[confused_dev]

I am trying to write some unittests using pytest, but I am coming accross the problem of how to mock my dbutils method when dbutils isn't being defined in my notebook.

Is there a way to do this so that I can unit test individual functions that are utilizing dbutils?

[fermin_vicente]

Hi,

You can mock dbutils. An easy way to do it is to:

• Try to receive dbutils as a parameter in your functions (inject it) instead of using it globally. This way your code is more testable and you won't to do patching which is a bit more cumbersome.
• Use a mock library. Unittest.mock is the simplest approach
• Call your function passing down a mock instead of the actual dbutils

Example:

• your library under test

def my_function(dbutils):
   ...
   dbutils.fs.ls("/tmp")  # this is using the local variable received by parameter
   ...

• your notebook

from my_library import my_function
 
...
my_function(dbutils)  # this refers to the global dbutils variable

• your test

from unittest.mock import MagicMock
 
from my_library import my_function
 
 
def test_my_function_calls_dbutils():
   mock_dbutils = MagicMock()
 
   my_function(dbutils=mock_dbutils)
 
   mock_dbutils.fs.ls.assert_called_once_with("/tmp")

[xiangzhu]

above reponse is a pure mock, and below is another example from dbx about a fixture with local filesystem, you can also add mock on dbutils.secrets:

https://github.com/databrickslabs/dbx/blob/b2989213b5f67e2b7ccf8adeba97da70e88ffff2/dbx/templates/pr...

[Anonymous]

Hi @Jake P​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!