โ08-27-2024 07:01 PM - edited โ08-27-2024 07:07 PM
Hi,
I am trying to install mosaic on my cluster, but get the error once I use 'enable_mosaic':
ImportError: cannot import name '_to_java_column' from 'pyspark.sql.functions' (/databricks/spark/python/pyspark/sql/functions/__init__.py)
File <command-1412689069782833>, line 2
1 from pyspark.sql.functions import *
----> 2 from mosaic import enable_mosaic
3 enable_mosaic(spark, dbutils)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/__init__.py:1
----> 1 from .api import *
2 from .core import MosaicLibraryHandler, MosaicContext
3 from .utils.display_handler import displayMosaic
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/__init__.py:1
----> 1 from .accessors import *
2 from .aggregators import *
3 from .constructors import *
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/accessors.py:2
1 from pyspark.sql import Column
----> 2 from pyspark.sql.functions import _to_java_column as pyspark_to_java_column
4 from mosaic.config import config
5 from mosaic.utils.types import ColumnOrName
The commands I used to import mosaic are:
%pip install databricks-mosaic
%python
from pyspark.sql.functions import *
from mosaic import enable_mosaic
enable_mosaic(spark, dbutils)
I have already verified that I have the dependencies required using
%python pip show databricks-mosaic
which outputs the following:
Name: databricks-mosaic
Version: 0.4.2
Summary: Mosaic: geospatial analytics in python, on Spark
Home-page: https://github.com/databrickslabs/mosaic
Author: Databricks
Author-email:
License: Databricks License
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages
Requires: geopandas, h3, ipython, keplergl, pyspark
Required-by:
My cluster configuration is as follows, with Photon enabled:
I've also tried to restart my cluster to no avail.
Is there a way I can get mosaic successfully enabled to access the spatial functions?
โ08-28-2024 03:34 AM
Hi @reggie ,
You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet all requirements:
Installation guide โ Mosaic (databrickslabs.github.io)
โ08-28-2024 03:32 AM
Hi @reggie ,
You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet with all reqeuirements:
Installation guide โ Mosaic (databrickslabs.github.io)
โ08-28-2024 03:34 AM
Hi @reggie ,
You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet all requirements:
Installation guide โ Mosaic (databrickslabs.github.io)
โ09-02-2024 05:46 PM
Thank you, I thought mosaic requires a DBR at least 13.x since I didn't get a version error, but this fixed my problem ๐
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group