https://community.databricks.com/t5/machine-learning/how-can-i-use-non-spark-related-libraries-like-spacy-with/m-p/24925#M1395 <P>It depends on what you mean, but if you're just trying to (say) tokenize and process data with spacy in parallel, then that's trivial. Write a 'pandas UDF' function that expresses how you want to transform data using spacy, in terms of a pandas DataFrame of input. Then you just apply that pandas UDF to your data with Spark; Spark will automatically chunk your data into pandas DataFrames, apply your function, and handle the results.</P> Thu, 17 Jun 2021 23:23:53 GMT sean_owen 2021-06-17T23:23:53Z https://community.databricks.com/t5/machine-learning/how-can-i-use-non-spark-related-libraries-like-spacy-with/m-p/24924#M1394 <P>I have an NLP application that I build on my local machine using spacy and pandas, but now I would like to scale my application to a large production dataset and utilize the benefits of sparks distributed compute. How do I import and utilize a library like spacy with Databricks/Spark?</P> Fri, 11 Jun 2021 18:55:55 GMT https://community.databricks.com/t5/machine-learning/how-can-i-use-non-spark-related-libraries-like-spacy-with/m-p/24924#M1394 User16752239203 2021-06-11T18:55:55Z https://community.databricks.com/t5/machine-learning/how-can-i-use-non-spark-related-libraries-like-spacy-with/m-p/24925#M1395 <P>It depends on what you mean, but if you're just trying to (say) tokenize and process data with spacy in parallel, then that's trivial. Write a 'pandas UDF' function that expresses how you want to transform data using spacy, in terms of a pandas DataFrame of input. Then you just apply that pandas UDF to your data with Spark; Spark will automatically chunk your data into pandas DataFrames, apply your function, and handle the results.</P> Thu, 17 Jun 2021 23:23:53 GMT https://community.databricks.com/t5/machine-learning/how-can-i-use-non-spark-related-libraries-like-spacy-with/m-p/24925#M1395 sean_owen 2021-06-17T23:23:53Z