Cannot use docker container with runtime 13.1 and above

[DaanRademaker]

Hi,

In our setup we use custom docker images on our job clusters. The setup of these images is in line with the docker images found on the databricks github page. The custom image works fine on runtimes, including 13.0. When using anything higher that 13.0 I get the following error for example when running spark.catalog.listTables()

```

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException

```

Very strange, almost seems like something is going wrong when databricks injects the magic runtime into the docker image and the java classpaths are not set correctly. I have looked at the changelog from runtime 13.0 to 13.1 but I cannot find why this should all of the sudden not work? Has anyone been able to get docker images to work on runtimes above 13.0?

The same happens both on an interactive cluster as well as a job cluster.

[Debayan]

Hi, Any specific error has been surfaced? 

Also, we need to check, Databricks Runtime for Machine Learning does not support Databricks Container Services.

Please tag @Debayan  with your next comment which will notify me. Thanks!

[DaanRademaker]

Hi @Debayan! Thanks for your reply.

If I attach the docker image to an interactive cluster and run spark.catalog.listTables("dbname")
I get the following error:
```

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/AlreadyExistsException

```

If I run the same command but attach the docker image to a job cluster I get the error mentioned above

```

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException

```

Both commands run fine if I attach docker image to a 13.0 cluster, but a 13.1 and 13.2 cluster don't work.

```

[Kaniz_Fatma]

The error message "java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/AlreadyExistsException" suggests a problem with the Hadoop Hive Metastore API.

The error message "java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException" suggests a problem with the Hive metadata.

To resolve this issue, checking the compatibility of the Spark and Hadoop versions being used is recommended. The version of Hadoop used in the 13.1 and 13.2 clusters may be incompatible with the Docker image being used. You can try upgrading the Hadoop version in the cluster or downgrading the Docker image to a version compatible with the Hadoop version in the cluster.

Additionally, you can check if any recent changes to the code or environment could have caused this error.

[DaanRademaker]

Thank you for your answer. However, I was under the impression that when using docker images in Databricks, Databricks actually injects the spark and hadoop packages into the image. This is how it seems to have always worked. The docker images are simply base ubuntu packages, nothing else. So in that case there must be going something wrong when these hadoop and spark packages are being injected.

[DaanRademaker · Accepted solution]

Also, this issue can be linked to the following github issue on the Databricks container images github page.
https://github.com/databricks/containers/issues/116

[DaanRademaker]

Interestingly enough I can use the exact same image on the 13.3LTS beta runtime without issues.

[DaanRademaker]

Interestingly enough I can use the exact same docker image on 13.3LTS Beta with no issues.