When we run DLT pipelines (which we deploy via DABs), we get a sporadic issue when attempting to install our bundle's wheel file.

First, in every DLT pipeline, we as a first step a script that looks like the following

import subprocess as sp
from importlib.util import find_spec as importlib_find_spec
from pathlib import Path

from databricks.sdk.runtime import dbutils, spark

if not importlib_find_spec('my_package'):
    artifact_path = spark.conf.get('bundle.artifactPath')

    if not artifact_path:
        msg = 'Bundle artifact path not found in Spark configuration.'
        raise ValueError(msg)

    wheel_path = next(
        (Path(artifact_path) / '.internal').glob(
            'my_package-*-py3-none-any.whl',
        ),
    )
    sp.check_call(  # noqa: S603
        [  # noqa: S607
            'pip',
            'install',
            str(wheel_path),
        ],
    )

    dbutils.library.restartPython()

Where we pass bundle.artifactPath in from the bundle in the pipeline options.

Some large percentage of the time, this works perfectly. From this point, we can run code which has available both our package (here called "my_package") as well as the 3rd party dependencies that are implied onto the .whl file by our project dependencies in the pyproject.toml.

However, every now and then, during the DLT pipeline "Initializing" stage, this will fail, with an error like the following

pyspark.errors.exceptions.captured.AnalysisException: Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-6584cd7b-1c47-4509-891d-918d38e18089/lib/python3.12/site-packages/my_package/base_pipeline_config.py", line 293, in _fcn
    return source.load()
           ^^^^^^^^^^^^^

pyspark.errors.exceptions.captured.AnalysisException: [PYTHON_DATA_SOURCE_ERROR] Failed to create Python data source instance: Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 192, in _read_with_length
    return self.loads(obj)
           ^^^^^^^^^^^^^^^
  File "/databricks/spark/python/pyspark/serializers.py", line 617, in loads
    return cloudpickle.loads(obj, encoding=encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'my_package'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 192, in _read_with_length
    return self.loads(obj)
           ^^^^^^^^^^^^^^^
  File "/databricks/spark/python/pyspark/serializers.py", line 617, in loads
    return cloudpickle.loads(obj, encoding=encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'my_package'

 SQLSTATE: 38000

Usually this ends up working after a re-run, though sometimes it can take several re-runs before it finally succeeds. And then sometimes it will work the first time several times in a row.

We'd really like to resolve this in a way that we can continue installing our wheel source code along with its external dependencies without this flakiness necessitating so many retries.

We thought about also trying to update sys.path, but this doesn't handle installing 3rd party dependencies. It seems like somehow the Python runtime gets out of sync.