How to load PDFs incrementally in volume?

[Niyojit]

I'm building an Intelligent Document Processing pipeline using Databricks AI Functions (ai_parse_document and ai_extract).

I want to ingest PDF files from a Google Drive folder into a Unity Catalog Volume. I can perform a full load successfully using the Google Drive connector, but I haven't found a way to incrementally load only new or modified PDF files into the Volume.

For structured files like CSV, COPY INTO supports incremental ingestion by tracking previously loaded files. However, I couldn't find an equivalent approach for PDF files when the destination is a Unity Catalog Volume.

does anyone have some workaround for it?

[balajij8 · Accepted solution]

You can use the Google Drive Managed connector that does most of it primarily built for syncing structured data directly to Delta Tables or the Standard connector that allows you to use Databricks (SQL, Auto Loader, pipelines) directly against a Google Drive URL using Unity Catalog connection.

You can follow below

• Incremental Standard Connector - You can use Lakeflow Connect Standard connector with Databricks SQL or Spark Structured Streaming and incrementally ingest the raw bytes of new PDFs directly into a Streaming Table. This bypasses the Unity Catalog Volume entirely and provides the exact incremental tracking you get with COPY INTO. You can parse the newly ingested binaries in a downstream materialized view or streaming table using AI functions. More details here

-- Read raw PDFs incrementally
CREATE OR REFRESH STREAMING TABLE raw AS
SELECT * 
FROM STREAM read_files(
  'gdrive',
  `databricks.connection` => 'drive_conn',
  format => 'binaryFile',
  pathGlobFilter => '*.pdf'
);​

CREATE OR REFRESH MATERIALIZED VIEW parsed_pdf AS 
SELECT 
  path,
  modificationTime,
  ai_parse_document(content) AS parsed_data
FROM raw;

• Incremental Managed Connector - You can use Managed connector with binaryFile ingestion option in Lakeflow Connect to process. More details here
• Auto Loader to Staging Volume - You can use Pyspark if you have a strict architectural requirement that the pdf files must stay inside a Unity Catalog Volume. You can combine the Lakeflow Connect Standard connector (via Auto Loader) with a PySpark foreachBatch operation. Auto Loader tracks which files in Google Drive are new, and standard Python file operations write the bytes out to the Volume.

def save_pdf_to_volume(row):
    import os
    # Extract filename from the Google Drive path
    file_name = os.path.basename(row.path)
    volume_path = f"/Volumes/my_volume/raw_pdfs/{file_name}"
    
    # Write the binary content to the Volume FUSE mount
    with open(volume_path, "wb") as f:
        f.write(row.content)

def process_batch(df, epoch_id):
    df.foreach(save_pdf_to_volume)

# 1. Use Auto Loader (cloudFiles) with the Lakeflow Google Drive connection
df = (spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "binaryFile")
  .option("databricks.connection", "gdrive_conn")
  .option("pathGlobFilter", "*.pdf")
  .load("gdrive")
)

# 2. Write each new file to the Volume and track state via checkpoint
(df.writeStream
  .foreachBatch(process_batch)
  .option("checkpointLocation", "/Volumes//my_volume/_checkpoints/pdf_ingest")
  .start()
)​