python 3.10 custom docker image

Erik_L — Tue, 07 Mar 2023 22:51:47 GMT

Goal: To use Python 3.10.4+

Why:

We have Python repos that are not backward compatible we want to use.

What:

I have created an image from Databricks example experimental containers already with Ubuntu 22.04 (2 major versions newer than the current) together with the updated Python version numbers, but otherwise kept everything the same.

When I run commands in Jupyter notebooks, I get the following errors. However, SQL and looking at the Data tab work fine with this compute. Why am I getting errors?

java.lang.Exception: Unable to start python kernel for ReplId-3848c-3c3c2-7bb73-1, kernel exited with exit code 1.
----- stdout -----
 
------------------
----- stderr -----
Traceback (most recent call last):
  File &quot;/databricks/python_shell/scripts/db_ipykernel_launcher.py&quot;, line 17, in &lt;module&gt;
    from dbruntime.MLWorkloadsInstrumentation import set_up_ml_workloads_instrumentation
  File &quot;/databricks/python_shell/dbruntime/MLWorkloadsInstrumentation/__init__.py&quot;, line 1, in &lt;module&gt;
    from dbruntime.dbutils import stderr
  File &quot;/databricks/python_shell/dbruntime/dbutils.py&quot;, line 6, in &lt;module&gt;
    from collections import namedtuple, Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
 
------------------
 
	at com.databricks.backend.daemon.driver.IpykernelUtils$.startReplFailure$1(JupyterDriverLocal.scala:1139)
	at com.databricks.backend.daemon.driver.IpykernelUtils$.$anonfun$startIpyKernel$3(JupyterDriverLocal.scala:1149)
	at com.databricks.backend.common.util.TimeUtils$.$anonfun$retryWithExponentialBackoff0$1(TimeUtils.scala:191)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at scala.util.Try$.apply(Try.scala:213)
	at com.databricks.backend.common.util.TimeUtils$.retryWithExponentialBackoff0(TimeUtils.scala:191)
	at com.databricks.backend.common.util.TimeUtils$.retryWithExponentialBackoff(TimeUtils.scala:145)
	at com.databricks.backend.common.util.TimeUtils$.retryWithTimeout(TimeUtils.scala:94)
	at com.databricks.backend.daemon.driver.IpykernelUtils$.startIpyKernel(JupyterDriverLocal.scala:1147)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal.$anonfun$startPython$1(JupyterDriverLocal.scala:855)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.Try$.apply(Try.scala:213)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal.com$databricks$backend$daemon$driver$JupyterDriverLocal$$withRetry(JupyterDriverLocal.scala:822)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal$$anonfun$com$databricks$backend$daemon$driver$JupyterDriverLocal$$withRetry$1.applyOrElse(JupyterDriverLocal.scala:825)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal$$anonfun$com$databricks$backend$daemon$driver$JupyterDriverLocal$$withRetry$1.applyOrElse(JupyterDriverLocal.scala:822)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
	at scala.util.Failure.recover(Try.scala:234)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal.com$databricks$backend$daemon$driver$JupyterDriverLocal$$withRetry(JupyterDriverLocal.scala:822)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal.startPython(JupyterDriverLocal.scala:839)
	at com.databricks.backend.daemon.driver.JupyterDriverLocal.&lt;init&gt;(JupyterDriverLocal.scala:535)
	at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:760)
	at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:349)
	at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:245)
	at java.lang.Thread.run(Thread.java:750)

Re: python 3.10 custom docker image

Erik_L — Tue, 07 Mar 2023 22:54:56 GMT

After searching for an hour, I realized what I needed to look for. It's the importing the Iterable from collections, which is deprecated in 3.10. I guess Databricks hasn't migrated code, yet. In which case, I'm at a cross-roads. Databricks 3.9, local code 3.10. Hm....

topic Re: python 3.10 custom docker image in Data Engineering

python 3.10 custom docker image

Re: python 3.10 custom docker image