Error with mosaic.enable_mosaic() when created DLT Pipeline with Mosaic lib
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 02:24 AM
The error was raised when I tried to start a DLT pipeline with simple code, just to start experimenting the DLT.
The primary library was Mosaic, which is already instructed to installed first before importing.
The code is roughly as follow
$ %pip install databricks-mosaic
import mosaic as mos
mos.enable_mosaic(spark, dbutils) # Error line
import dlt
from pyspark.sql.functions import *
from pyspark.sql.types import *
@dlt.table(comment="Testing a DLT table for geospatial coverage")
def geo_area():
return spark.read.table("geo_coverage_area")
Below is the DLT Pipeline setting JSON
{
"id": "<>",
"pipeline_type": "WORKSPACE",
"clusters": [
{
"label": "default",
"node_type_id": "m5d.large",
"driver_node_type_id": "m5d.large",
"custom_tags": {
"type": "test"
},
"num_workers": 1
},
{
"label": "maintenance",
"custom_tags": {
"type": "test"
}
}
],
"development": true,
"continuous": false,
"channel": "CURRENT",
"photon": false,
"libraries": [
{
"notebook": {
"path": "/<>/test DLT"
}
}
],
"name": "helloCellCoverage",
"edition": "CORE",
"catalog": "sigint_workspace",
"target": "default",
"data_sampling": false
}
And here is the error raised
java.lang.RuntimeException: Failed to execute python command for notebook "<Removed>" and error AnsiResult(---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
File <command--1>:3
1 import mosaic as mos
----> 3 mos.enable_mosaic(spark, dbutils)
5 import dlt
6 # import pyspark.sql.functions as
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2024 09:50 PM
Hi, thank you for the reply. However, I couldn't related the OAuth issue to the Java RunTime error from my question. Additionally, I tried checked the Photon feature when started the workflow, and the error still persists.
Thank you for your response anyway.