Re: Using pyspark databricks UDFs with outside fun...

tp992 · ‎03-07-2025

I think the solution is in .addArtifact if I read this:

But have not gotten it to work just yet.

```

spark = DatabricksSession.builder.getOrCreate()

venv_pack.pack(output='pyspark_venv.tar.gz')

spark.addArtifact(

"pyspark_venv.tar.gz#environment",

archive=True)

spark.conf.set(

"spark.sql.execution.pyspark.python", "environment/bin/python")

spark.addArtifact("minimal_example_1/module/udf.py", pyfile=True)

```