- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 07:50 AM - edited 07-28-2023 07:51 AM
Hi,
In our setup we use custom docker images on our job clusters. The setup of these images is in line with the docker images found on the databricks github page. The custom image works fine on runtimes, including 13.0. When using anything higher that 13.0 I get the following error for example when running spark.catalog.listTables()
```
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException
```
Very strange, almost seems like something is going wrong when databricks injects the magic runtime into the docker image and the java classpaths are not set correctly. I have looked at the changelog from runtime 13.0 to 13.1 but I cannot find why this should all of the sudden not work? Has anyone been able to get docker images to work on runtimes above 13.0?
The same happens both on an interactive cluster as well as a job cluster.