The error was raised when I tried to start a DLT pipeline with simple code, just to start experimenting the DLT.
The primary library was Mosaic, which is already instructed to installed first before importing.
The code is roughly as follow
$ %pip install databricks-mosaic
import mosaic as mos
mos.enable_mosaic(spark, dbutils) # Error line
import dlt
from pyspark.sql.functions import *
from pyspark.sql.types import *
@dlt.table(comment="Testing a DLT table for geospatial coverage")
def geo_area():
return spark.read.table("geo_coverage_area")
Below is the DLT Pipeline setting JSON
{
"id": "<>",
"pipeline_type": "WORKSPACE",
"clusters": [
{
"label": "default",
"node_type_id": "m5d.large",
"driver_node_type_id": "m5d.large",
"custom_tags": {
"type": "test"
},
"num_workers": 1
},
{
"label": "maintenance",
"custom_tags": {
"type": "test"
}
}
],
"development": true,
"continuous": false,
"channel": "CURRENT",
"photon": false,
"libraries": [
{
"notebook": {
"path": "/<>/test DLT"
}
}
],
"name": "helloCellCoverage",
"edition": "CORE",
"catalog": "sigint_workspace",
"target": "default",
"data_sampling": false
}
And here is the error raised
java.lang.RuntimeException: Failed to execute python command for notebook "<Removed>" and error AnsiResult(---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
File <command--1>:3
1 import mosaic as mos
----> 3 mos.enable_mosaic(spark, dbutils)
5 import dlt
6 # import pyspark.sql.functions as
