Issue enabling mosaic

reggie — Wed, 28 Aug 2024 02:07:57 GMT

Hi,

I am trying to install mosaic on my cluster, but get the error once I use 'enable_mosaic':

ImportError: cannot import name '_to_java_column' from 'pyspark.sql.functions' (/databricks/spark/python/pyspark/sql/functions/__init__.py) File <command-1412689069782833>, line 2 1 from pyspark.sql.functions import * ----> 2 from mosaic import enable_mosaic 3 enable_mosaic(spark, dbutils) File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/__init__.py:1 ----> 1 from .api import * 2 from .core import MosaicLibraryHandler, MosaicContext 3 from .utils.display_handler import displayMosaic File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/__init__.py:1 ----> 1 from .accessors import * 2 from .aggregators import * 3 from .constructors import * File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/accessors.py:2 1 from pyspark.sql import Column ----> 2 from pyspark.sql.functions import _to_java_column as pyspark_to_java_column 4 from mosaic.config import config 5 from mosaic.utils.types import ColumnOrName

The commands I used to import mosaic are:

%pip install databricks-mosaic

%python from pyspark.sql.functions import * from mosaic import enable_mosaic enable_mosaic(spark, dbutils)

I have already verified that I have the dependencies required using

%python pip show databricks-mosaic

which outputs the following:

Name: databricks-mosaic Version: 0.4.2 Summary: Mosaic: geospatial analytics in python, on Spark Home-page: https://github.com/databrickslabs/mosaic Author: Databricks Author-email: License: Databricks License Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages Requires: geopandas, h3, ipython, keplergl, pyspark Required-by:

My cluster configuration is as follows, with Photon enabled:

2-5 Workers 256-640 GB
Memory 64-160 Cores
1 Driver128 GB Memory, 32 Cores
Runtime14.3.x-scala2.12
Unity Catalog
Photon
Standard_D32a_v4
36–72 DBU/h

I've also tried to restart my cluster to no avail.

Is there a way I can get mosaic successfully enabled to access the spatial functions?

Re: Issue enabling mosaic

szymon_dybczak — Wed, 28 Aug 2024 10:32:11 GMT

Hi @reggie ,

You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet with all reqeuirements:

Installation guide — Mosaic (databrickslabs.github.io)

Re: Issue enabling mosaic

szymon_dybczak — Wed, 28 Aug 2024 10:34:14 GMT

Hi @reggie ,

You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet all requirements:

Installation guide — Mosaic (databrickslabs.github.io)

Re: Issue enabling mosaic

reggie — Tue, 03 Sep 2024 00:46:43 GMT

Thank you, I thought mosaic requires a DBR at least 13.x since I didn't get a version error, but this fixed my problem 🙂

Re: Issue enabling mosaic

JuanRomero — Thu, 13 Feb 2025 20:43:34 GMT

Thanks, I had the same issue and I was able to fix it

Re: Issue enabling mosaic

KyleL — Wed, 02 Jul 2025 00:47:13 GMT

Hello all, do we know when databricks-mosaic will get an update support the latest spark/scala runtime?

Thanks

topic Issue enabling mosaic in Get Started Discussions

Issue enabling mosaic

Re: Issue enabling mosaic

Re: Issue enabling mosaic

Re: Issue enabling mosaic

Re: Issue enabling mosaic

Re: Issue enabling mosaic