@AFH so in the end we ended up finding a "solution"  to the notebook issue. This issue occurs in "serverless". The only way to make it work for validation is by spinning up a compute and connecting your notebook to it.

In the context of DLT pipelines, we actually hit this issue as well. We tried to install our DAB via a initial "install" job, however that ended up with race conditions. We initially wanted to install the wheel via `libraries.whl` on the yml specification, but decided against it since that was a deprecated option.

After a while I managed to find https://docs.databricks.com/api/workspace/pipelines/create#environment . So ultimately our pipelines look like:

resources:
  pipelines:
    my_pipeline_name:
        .....
        environment:
          dependencies:
             - '/path/to/where/dab/is/built/and/loaded/in/unity/catalog'

This apparently works with wild cards, so for example i could specify `- '/my/path/my_cool_wheel_*_py3-none-any.whl'` and it managed to install it. After this our pipelines are using our custom structured streaming reliably.