GDAL on Databricks Cluster Runtime 12.2 LTS

[NC]

I need gdal in my course work.

After reading this post, I used init script as follows to install gdal into runtime 12.2 LTS

dbutils.fs.put("/databricks/scripts/gdal_install.sh","""
#!/bin/bash
sudo add-apt-repository ppa:ubuntugis/ppa
sudo apt-get update
sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal""",
True)

The init script ran and cluster could start properly but when i run import gdal in notebook, i get the following error:

ModuleNotFoundError: No module named 'gdal'

I also tried installing gdal into the cluster via Maven repository, it does not work either.

May I know what I can do to get gdal installed properly?

Thank you.

[Debayan]

Hi, Could you please confirm if apt get install gdal was already tried? 

You can try as below: 

apt-get update && apt-get upgrade; apt-get install gdal*;

Please tag @Debayan with your next comment, which will get me notified. Thanks!

[NC]

Hi @Debayan, 

Do you mean this:?

dbutils.fs.put("/databricks/scripts/gdal_install.sh","""
#!/bin/bash
apt-get update && apt-get upgrade
apt-get install gdal*
""",
True)

[Debayan]

Hi, I think I understood the issue, for the no module found error sometimes you need to clean the apt list , something like below: 

%sh
rm -r /var/cache/apt/archives/* /var/lib/apt/lists/*
sudo apt-get clean
sudo apt-get update
sudo add-apt-repository -y ppa:ubuntugis/ppa && apt-get update
sudo apt-get install -y gdal-bin libgdal-dev (or gdal*)

[Matt_C]

Hi, in case anyone is still struggling here. I found I could not get the init script approach to work, but if I just run a shell command to install gdal at the start of my notebook it works fine. You might note, however, that this installs gdal version 3.4.1 which is a bit dated (Jan 4, 2022). My understanding is that this is because "the Databricks standard image is itself based on Ubuntu 18.04 and GDAL 2.2.3 is the latest version available on this distribution" - see this Medium article. 

%sh
sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal

%python
from osgeo import gdal
gdal.__version__