Just use an iterator of Series UDF. See here: https://docs.databricks.com/spark/latest/spark-sql/udf-python-pandas.html#iterator-of-series-to-iter...
This allows you to set up some pre-defined state (like loading a file) before doing the computation.
