PDF Parsing in Notebook

[Kamal2]

I have pdf files stored in azure adls.

i want to parse pdf files in pyspark dataframes

how can i do that ?

[-werners-]

I know of Apache Tika. But that is a java lib and I do not know if there are python bindings.

Pypi has a python version though:

https://pypi.org/project/tika/

It might help.

[morganmazouchi · Accepted solution]

If you have familiarity with Scala you can use Tika. Tika is a wrapper around PDFBox. In case you want to use it in Databricks I suggest you to go through this blog and Git repo. For python based codes you may want to use PyPDF2 as a pandas UDF in Spark.

[Mykola_Melnyk]

Please look to the PDF DataSource for Apache Spark.

This project provides a custom data source for the Apache Spark that allows you to read PDF files into the Spark DataFrame. And here notebook with example of usage.

df = spark.read.format("pdf") \
    .option("imageType", "BINARY") \
    .option("resolution", "200") \
    .option("pagePerPartition", "2") \
    .option("reader", "pdfBox") \
    .load("path to the pdf file(s)")

df.show()

[Mykola_Melnyk]

PDF Data Source works now on Databricks.
Instruction with example: https://stabrise.com/blog/spark-pdf-on-databricks/

[Mykola_Melnyk]

Spark PDF works now with Unity Catalog volumes, started from 0.1.16 version: more details here: https://stabrise.com/blog/spark-pdf-databricks-unity-catalog/