How to load PDFs incrementally in volume?

Niyojit — Mon, 20 Jul 2026 12:33:45 GMT

I'm building an Intelligent Document Processing pipeline using Databricks AI Functions (ai_parse_document and ai_extract).

I want to ingest PDF files from a Google Drive folder into a Unity Catalog Volume. I can perform a full load successfully using the Google Drive connector, but I haven't found a way to incrementally load only new or modified PDF files into the Volume.

For structured files like CSV, COPY INTO supports incremental ingestion by tracking previously loaded files. However, I couldn't find an equivalent approach for PDF files when the destination is a Unity Catalog Volume.

does anyone have some workaround for it?

Re: How to load PDFs incrementally in volume?

balajij8 — Mon, 20 Jul 2026 14:30:57 GMT

You can use the Google Drive Managed connector that does most of it primarily built for syncing structured data directly to Delta Tables or the Standard connector that allows you to use Databricks (SQL, Auto Loader, pipelines) directly against a Google Drive URL using Unity Catalog connection.

You can follow below

Incremental Standard Connector - You can use Lakeflow Connect Standard connector with Databricks SQL or Spark Structured Streaming and incrementally ingest the raw bytes of new PDFs directly into a Streaming Table. This bypasses the Unity Catalog Volume entirely and provides the exact incremental tracking you get with COPY INTO. You can parse the newly ingested binaries in a downstream materialized view or streaming table using AI functions. More details here

-- Read raw PDFs incrementally CREATE OR REFRESH STREAMING TABLE raw AS SELECT * FROM STREAM read_files( 'gdrive', `databricks.connection` => 'drive_conn', format => 'binaryFile', pathGlobFilter => '*.pdf' ); CREATE OR REFRESH MATERIALIZED VIEW parsed_pdf AS SELECT path, modificationTime, ai_parse_document(content) AS parsed_data FROM raw;

Incremental Managed Connector - You can use Managed connector with binaryFile ingestion option in Lakeflow Connect to process. More details here
Auto Loader to Staging Volume - You can use Pyspark if you have a strict architectural requirement that the pdf files must stay inside a Unity Catalog Volume. You can combine the Lakeflow Connect Standard connector (via Auto Loader) with a PySpark foreachBatch operation. Auto Loader tracks which files in Google Drive are new, and standard Python file operations write the bytes out to the Volume.

def save_pdf_to_volume(row): import os # Extract filename from the Google Drive path file_name = os.path.basename(row.path) volume_path = f"/Volumes/my_volume/raw_pdfs/{file_name}" # Write the binary content to the Volume FUSE mount with open(volume_path, "wb") as f: f.write(row.content) def process_batch(df, epoch_id): df.foreach(save_pdf_to_volume) # 1. Use Auto Loader (cloudFiles) with the Lakeflow Google Drive connection df = (spark.readStream .format("cloudFiles") .option("cloudFiles.format", "binaryFile") .option("databricks.connection", "gdrive_conn") .option("pathGlobFilter", "*.pdf") .load("gdrive") ) # 2. Write each new file to the Volume and track state via checkpoint (df.writeStream .foreachBatch(process_batch) .option("checkpointLocation", "/Volumes//my_volume/_checkpoints/pdf_ingest") .start() )

topic Re: How to load PDFs incrementally in volume? in Data Engineering

How to load PDFs incrementally in volume?

Re: How to load PDFs incrementally in volume?