I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error:
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
Traceback (most recent call last):
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 18, in <module>
from dbruntime.MLWorkloadsInstrumentation import set_up_ml_workloads_instrumentation
File "/databricks/python_shell/dbruntime/MLWorkloadsInstrumentation/__init__.py", line 1, in <module>
from dbruntime.dbutils import stderr
File "/databricks/python_shell/dbruntime/dbutils.py", line 6, in <module>
from collections import namedtuple, Iterable
I also noticed that DBR's UI reports the compute as "Single node: m5.large · Spot · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · auto". Note that the Compute has v11.3-LTS, although my Spark workflow uses a container with version 13.3-LTS. This version mismatch seems odd.
Googling around, I see it looks like there's some nuance to Python version (https://github.com/tctianchi/pyvenn/issues/19). Apparently the "from collections import Iterable" is supposed to be changed to "from collections.abc import Iterable". The code itself is coming from DBR, not ours, however.
I suspect there's some problem in how I've configured the job; any pointers on version/container selection are appreciated.
