topic Re: Rasterio on shared/standard cluster has no access to proj.db in Data Engineering

Rasterio on shared/standard cluster has no access to proj.db

der — Wed, 22 Oct 2025 13:01:00 GMT

We try to use rasterio on a Databricks shared/standard cluster with DBR 17.1. Rasterio is directly installed on the cluster as library.

Code:

import rasterio rasterio.show_versions()

Output:

rasterio info:
rasterio: 1.4.3
GDAL: 3.9.3
PROJ: 9.4.1
GEOS: 3.11.1
PROJ DATA: /databricks/native/proj-data
GDAL DATA: /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/rasterio/gdal_data

System:
python: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0]
executable: /local_disk0/.ephemeral_nfs/envs/pythonEnv-b2347f39-219b-43b3-a5db-676ce38e43ca/bin/python
machine: Linux-5.15.0-1092-azure-x86_64-with-glibc2.39

Python deps:
affine: 2.4.0
attrs: 24.3.0
certifi: 2025.01.31
click: 8.1.7
cligj: 0.7.2
cython: 3.0.12
numpy: 2.1.3
click-plugins: None
setuptools: 74.0.0

Test script:

import numpy as np from rasterio.io import MemoryFile from rasterio.transform import from_origin meta = { "driver": "GTiff", "height": 1, "width": 1, "count": 1, "dtype": "uint8", "crs": "EPSG:2056", # <-- forces an EPSG lookup in proj.db "transform": from_origin(0, 1, 1, 1), } with MemoryFile() as mem: with mem.open(**meta) as ds: ds.write(np.zeros((1, 1, 1), dtype="uint8"))

CRSError: The EPSG code is unknown. PROJ: internal_proj_create_from_database: Cannot find proj.db

So we have no access to /databricks/native/proj-data where the proj.db is stored.

I am aware, that there are possible hacks and workarounds:

putting the proj.db somewhere where users have access and than work with environment variables as PROJ_DATA
write the test script that no lookup is needed

However, this is just working around the issue. Dear Databricks could you please change this access rights or give me more insights, why it is the right way?

Re: Rasterio on shared/standard cluster has no access to proj.db

szymon_dybczak — Wed, 22 Oct 2025 13:29:41 GMT

Hi @der,

I guess this is related to limitation of standard/shared cluster access mode which you're using.

https://docs.databricks.com/aws/en/compute/standard-limitations#network-and-file-system-limitations

You can try to go with dedicated access mode or is that's not an option for you then setup path to proj.db to i.e UC Volume which should be accessible by standard access mode.

Re: Rasterio on shared/standard cluster has no access to proj.db

der — Wed, 22 Oct 2025 14:57:52 GMT

Hi @szymon_dybczak

Yes on dedicated access mode it works fine, because we have access to /databricks/native/proj-data. From a cost perspective we want to stay on standard/shared cluster.

Pushing proj.db to UC volume and change the environment path is what I meant with hack/workaround 1.

I still do not get it why the "proj-data" is in a path, where we have no "read" access from a standard cluster?

Re: Rasterio on shared/standard cluster has no access to proj.db

Chiran-Gajula — Wed, 22 Oct 2025 15:16:07 GMT

Hi @der

Can you try adding this in your test script.

import os

os.environ["PROJ_LIB"]="/databricks/native/proj-data"

Hope users have access to this path /databricks/native/proj-data

Re: Rasterio on shared/standard cluster has no access to proj.db

szymon_dybczak — Wed, 22 Oct 2025 15:21:56 GMT

This is explained in the limitations section in my previous answer.

Re: Rasterio on shared/standard cluster has no access to proj.db

der — Wed, 22 Oct 2025 15:22:41 GMT

Hi @Chiran-Gajula,

Exactly this is the issue. On standard cluster there is no access to /databricks/native/proj-data where the proj.db is stored.

So your proposed solution won't work.

Re: Rasterio on shared/standard cluster has no access to proj.db

szymon_dybczak — Wed, 22 Oct 2025 15:54:00 GMT

To make it easier:

"Standard compute runs commands as a low-privilege user forbidden from accessing sensitive parts of the filesystem.

POSIX-style paths (/) for DBFS are not supported."

Re: Rasterio on shared/standard cluster has no access to proj.db

der — Wed, 22 Oct 2025 15:54:13 GMT

Which limitations do you mean?

Standard compute runs commands as a low-privilege user forbidden from accessing sensitive parts of the filesystem.

Not sure if proj-data is sensitive data.

So for Databricks would be simple to change the group to "spark-users" and change the rights.

sudo chown -R root:spark-users native
sudo chmod -R 750 native
sudo chmod g+s native

No workaround would be needed. They did the same for "licenses", "python3", ....

Re: Rasterio on shared/standard cluster has no access to proj.db

szymon_dybczak — Wed, 22 Oct 2025 16:22:38 GMT

Exactly that part. In shared access mode they specifically forbidden access to some paths. Of course you can complain about this but they won't change this behavior because your library doesn't work as expected - especially since other approaches - like using Volumes or other access mode are available.

Re: Rasterio on shared/standard cluster has no access to proj.db

der — Thu, 06 Nov 2025 10:15:10 GMT

@szymon_dybczak

Yes, we could replicate the content of /databricks/native/proj-data to a volume, cloud-native storage, dbfs (which we have deactivated by the way) or somewhere else. But after that, we have to change the environment variable PROJ_DATA pointing to the new location. If we change the environment variable, we introduce new issues. Guess what the native Databricks geospatial function relies on the same environment variable and do not work anymore correctly!

And the library (rasterio) use PROJ system library as it should. Additional Information about rasterio and PROJ_DATA

https://rasterio.readthedocs.io/en/stable/faq.html#why-can-t-rasterio-find-proj-db-rasterio-from-pypi-versions-1-2-0

Re: Rasterio on shared/standard cluster has no access to proj.db

der — Thu, 06 Nov 2025 10:29:18 GMT

Current Workaround:
If you select the "Photon" engine on a Standard/Shared Cluster, they change the access rights of /databricks/native/proj-data and rasterio works fine.

The downside:
Pay for "Photon" compute to use a Python library, which do not use Spark/Photon engine -> $$$

Future:
I talked to Databricks support and they will check if the engineering team can activate the same for Standard engine with Standard/Shared access mode, because the folder content is definitely not sensitive.