https://community.databricks.com/t5/machine-learning/what-are-best-nlp-libraries-to-use-with-spark/m-p/24033#M1354 <P>Best NLP APIs to use with Spark which gives better performance</P> Wed, 16 Jun 2021 12:49:23 GMT Srikanth_Gupta_ 2021-06-16T12:49:23Z https://community.databricks.com/t5/machine-learning/what-are-best-nlp-libraries-to-use-with-spark/m-p/24033#M1354 <P>Best NLP APIs to use with Spark which gives better performance</P> Wed, 16 Jun 2021 12:49:23 GMT https://community.databricks.com/t5/machine-learning/what-are-best-nlp-libraries-to-use-with-spark/m-p/24033#M1354 Srikanth_Gupta_ 2021-06-16T12:49:23Z https://community.databricks.com/t5/machine-learning/what-are-best-nlp-libraries-to-use-with-spark/m-p/24034#M1355 <P>By far the most popular and comprehensive library, to my knowledge, for Spark-native distributed NLP, is spark-nlp from John Snow Labs. <A href="https://nlp.johnsnowlabs.com/" target="test_blank">https://nlp.johnsnowlabs.com/</A> It is open source (but with commercial support options) and has a whole lot of functionality.</P><P></P><P>You can also use spacy, nltk, and other non-Spark NLP libraries with Spark, by writing pandas UDFs that leverage these libraries, then applying them to data with Spark.</P> Thu, 17 Jun 2021 19:59:25 GMT https://community.databricks.com/t5/machine-learning/what-are-best-nlp-libraries-to-use-with-spark/m-p/24034#M1355 sean_owen 2021-06-17T19:59:25Z