In a Databricks notebook, I need to run text files (stdin, stdout) through a function from an external library. I have used sparkContext.AddFiles({external_library_name}) to add the external library so that it is available to all executors.
when I run sparkFiles.get({external_library_name}) it returns the executor path to the external library. When I use that sparkFiles.get({external_library_name}) location as part of an Rdd.Pipe call with concatenated params, I get a FileNotFound exception.
spark.sparkContext.addFile("/dbfs/FileStore/Custom_Executable")
files_rdd = spark.sparkContext.parallelize(files_list)
print(f'spark file path:{SparkFiles.get("Custom_Executable")}')
path_with_params = SparkFiles.get("Custom_Executable") + " the-function-name --to
company1 --from company2 -input - -output -"
print(f'path with params: {path_with_params}')
pipe_rdd = files_rdd.pipe(path_with_params, env={'SOME_ENV_VAR': env_var_val})
print(pipe_tokenised_rdd.collect())
The output from this
spark file path:/local_disk0/spark-c69e5328-9da3-4c76-85b8-a977e470909d/userFiles-
e8a37109-046c-4909-8dd2-95bde5c9f3e3/Custom_Executable
exe path: /local_disk0/spark-c69e5328-9da3-4c76-85b8-a977e470909d/userFiles-e8a37109-
046c-4909-8dd2-95bde5c9f3e3/Custom_Executable the-function-name --to company1 --from
company2 -input - -output -
Output
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage
0.0 failed 4 times, most recent failure: Lost task 2.3 in stage 0.0 (TID 12)
(10.113.4.168 executor 0): org.apache.spark.api.python.PythonException:
'FileNotFoundError: [Errno 2] No such file or directory: '/local_disk0/spark-
c69e5328-9da3-4c76-85b8-a977e470909d/userFiles-e8a37109-046c-4909-8dd2-
95bde5c9f3e3/Custom_Executable''. Full traceback below:
Why is the pipe call not finding the location returned by SparkFiles.get?
