topic Re: GDAL on Databricks Cluster Runtime 12.2 LTS in Get Started Discussions

GDAL on Databricks Cluster Runtime 12.2 LTS

NC — Fri, 04 Aug 2023 10:01:20 GMT

I need gdal in my course work.

After reading this post, I used init script as follows to install gdal into runtime 12.2 LTS

dbutils.fs.put("/databricks/scripts/gdal_install.sh",""" #!/bin/bash sudo add-apt-repository ppa:ubuntugis/ppa sudo apt-get update sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal""", True)

The init script ran and cluster could start properly but when i run import gdal in notebook, i get the following error:

ModuleNotFoundError: No module named 'gdal'

I also tried installing gdal into the cluster via Maven repository, it does not work either.

May I know what I can do to get gdal installed properly?

Thank you.

Re: GDAL on Databricks Cluster Runtime 12.2 LTS

Debayan — Mon, 21 Aug 2023 07:36:02 GMT

Hi, Could you please confirm if apt get install gdal was already tried?

You can try as below:

apt-get update && apt-get upgrade; apt-get install gdal*;

Please tag @Debayan with your next comment, which will get me notified. Thanks!

Re: GDAL on Databricks Cluster Runtime 12.2 LTS

NC — Mon, 21 Aug 2023 14:26:16 GMT

Hi @Debayan,

Do you mean this:?

dbutils.fs.put("/databricks/scripts/gdal_install.sh","""
#!/bin/bash
apt-get update && apt-get upgrade
apt-get install gdal*
""",
True)

Re: GDAL on Databricks Cluster Runtime 12.2 LTS

Debayan — Tue, 22 Aug 2023 06:52:31 GMT

Hi, I think I understood the issue, for the no module found error sometimes you need to clean the apt list , something like below:

%sh
rm -r /var/cache/apt/archives/* /var/lib/apt/lists/*
sudo apt-get clean
sudo apt-get update
sudo add-apt-repository -y ppa:ubuntugis/ppa && apt-get update
sudo apt-get install -y gdal-bin libgdal-dev (or gdal*)

Re: GDAL on Databricks Cluster Runtime 12.2 LTS

Matt_C — Fri, 01 Nov 2024 16:47:17 GMT

Hi, in case anyone is still struggling here. I found I could not get the init script approach to work, but if I just run a shell command to install gdal at the start of my notebook it works fine. You might note, however, that this installs gdal version 3.4.1 which is a bit dated (Jan 4, 2022). My understanding is that this is because "the Databricks standard image is itself based on Ubuntu 18.04 and GDAL 2.2.3 is the latest version available on this distribution" - see this Medium article.

%sh sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal

%python from osgeo import gdal gdal.__version__