All in all, I got confused by a misconfiguration.
A notification was setup as a webhook notificaton instead of a slack notification by mistake in a particular workspace, which caused the confusion. So, "problem in chair, not computer".
If anything, it would be to provide more of the error message in the test output, which could make it easier to spot the mistake faster, and I suppose also to debug any issues with webhook notifications.