https://community.databricks.com/t5/data-engineering/optimal-approach-when-using-external-script-executable-for/m-p/17722#M11685 <P>I need to process a number of files where I manipulate file text utilising an external executable that operates on stdin/stdout.</P><P> I am quite new to spark. What I am attempting is to use rdd.pipe as in the following</P><PRE><CODE>exe_path = " /usr/local/bin/external-exe" files_rdd = spark.sparkContext.parallelize(files_list) pipe_tokenised_rdd = files_rdd.pipe(exe_path}) pipe_tokenised_rdd.collect())</CODE></PRE><P>Is this approach using rdd.pipe the way that external code should be used in general? Do I need to use rdd.pipe or should I use dataframe with transform. Looking for advice on approaches.</P> Tue, 14 Jun 2022 13:04:25 GMT mick042 2022-06-14T13:04:25Z https://community.databricks.com/t5/data-engineering/optimal-approach-when-using-external-script-executable-for/m-p/17722#M11685 <P>I need to process a number of files where I manipulate file text utilising an external executable that operates on stdin/stdout.</P><P> I am quite new to spark. What I am attempting is to use rdd.pipe as in the following</P><PRE><CODE>exe_path = " /usr/local/bin/external-exe" files_rdd = spark.sparkContext.parallelize(files_list) pipe_tokenised_rdd = files_rdd.pipe(exe_path}) pipe_tokenised_rdd.collect())</CODE></PRE><P>Is this approach using rdd.pipe the way that external code should be used in general? Do I need to use rdd.pipe or should I use dataframe with transform. Looking for advice on approaches.</P> Tue, 14 Jun 2022 13:04:25 GMT https://community.databricks.com/t5/data-engineering/optimal-approach-when-using-external-script-executable-for/m-p/17722#M11685 mick042 2022-06-14T13:04:25Z https://community.databricks.com/t5/data-engineering/optimal-approach-when-using-external-script-executable-for/m-p/17723#M11686 <P>Hi @Michael Lennon​&nbsp; Can you please elaborate use case on what the external app is doing exe_path</P> Fri, 09 Sep 2022 15:21:26 GMT https://community.databricks.com/t5/data-engineering/optimal-approach-when-using-external-script-executable-for/m-p/17723#M11686 User16753725469 2022-09-09T15:21:26Z