Using GraphFrames on DLT job

lprevost — Tue, 13 Aug 2024 14:30:41 GMT

I am trying to run a DLT job that uses GraphFrames, which is in the ML standard image. I am using it successfully in my job compute instances. Here are my overrides for the standard job compute policy:

{
"spark_version": {
"type": "unlimited",
"defaultValue": "auto:latest-lts-ml"
},
"cluster_type": {
"type": "allowlist",
"defaultValue": "all-purpose",
"values": [
"all-purpose",
"job",
"dlt"
]
}

}

However, when I run the DLT job, I get the following error:

ModuleNotFoundError: No module named 'graphframes',None,Map(),Map(),List(),List(),Map())

Will DLT pipelines support GraphFrames?

Re: Using GraphFrames on DLT job

lprevost — Sat, 21 Sep 2024 18:26:21 GMT

Crickets ....

topic Re: Using GraphFrames on DLT job in Data Engineering

Using GraphFrames on DLT job

Re: Using GraphFrames on DLT job