Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration

Chris_Shehu — Fri, 21 Oct 2022 13:17:42 GMT

I'm currently trying to follow the Software engineering best practices for notebooks - Azure Databricks guide but I keep running into the following during step 4.5: Run the test

============================= test session starts ==============================

platform linux -- Python 3.9.5, pytest-7.1.2, pluggy-1.0.0

rootdir: /Workspace/Repos/CHNW/Development

collected 0 items / 1 error

==================================== ERRORS ====================================

__________________ ERROR collecting tests/transforms_test.py ___________________

/usr/lib/python3.9/importlib/__init__.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

<frozen importlib._bootstrap>:1030: in _gcd_import

???

<frozen importlib._bootstrap>:1007: in _find_and_load

???

<frozen importlib._bootstrap>:986: in _find_and_load_unlocked

???

<frozen importlib._bootstrap>:680: in _load_unlocked

???

/local_disk0/.ephemeral_nfs/envs/pythonEnv-d47fc3aa-b110-485d-9084-2b0093f9502e/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:159: in exec_module

source_stat, co = _rewrite_test(fn, self.config)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-d47fc3aa-b110-485d-9084-2b0093f9502e/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:357: in _rewrite_test

source = fn.read_bytes()

/usr/lib/python3.9/pathlib.py:1249: in read_bytes

with self.open(mode='rb') as f:

/usr/lib/python3.9/pathlib.py:1242: in open

return io.open(self, mode, buffering, encoding, errors, newline,

/usr/lib/python3.9/pathlib.py:1110: in _opener

return self._accessor.open(self, flags, mode)

E OSError: [Errno 95] Operation not supported: '/Workspace/Repos/CHNW/Development/tests/transforms_test.py'

=========================== short test summary info ============================

ERROR tests/transforms_test.py - OSError: [Errno 95] Operation not supported:...

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

=============================== 1 error in 1.19s ===============================

I'm running a 11+ cluster for this process and have confirmed multiple times that my setup matches the setup in the documentation.

I'm not sure what I'm missing as this is mostly copy/paste

Re: Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration

Chris_Shehu — Mon, 24 Oct 2022 18:09:38 GMT

Closing the loop on this in case anyone gets stuck in the same situation. You can see in the images that the transforms_test.py shows a different icon then the testdata.csv. This is because it was saved as a juypter notebook not a .py file. When the script runs it's trying to access the file but breaks because it can't use the notebook. Re-creating the file as a .py fixed the issue.

topic Re: Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration in Data Engineering

Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration

Re: Getting errors while following Microsoft Databricks Best-Practices for DevOps Integration