Databricks Community

kapil_dua · Tuesday

Data Ingestion (Lakeflow) Feature Bug:

While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default data type as String, which gives error as Data Type mismatch.

This bug needs to be fixed by Databricks

AbhilashNagilla · Tuesday

Reading the PDFs with the binaryFile format returns a fixed schema (path, modificationTime, length, content) with content already typed as BINARY, so there's no column type to set (binary file data source, read_files). You can run this in the SQL editor on a pro or serverless SQL warehouse (requirements) :

CREATE OR REFRESH STREAMING TABLE <catalog>.<schema>.pdf_raw
AS SELECT * FROM STREAM read_files(
  '/Volumes/<catalog>/<schema>/<volume>/<folder>/',
  format => 'binaryFile',
  fileNamePattern => '*.{pdf,PDF}'
);

Add SCHEDULE EVERY 1 hour after the table name if you want scheduled refreshes (standalone streaming tables).

Jen_Evangelista · yesterday

hI, @kapil_dua., to isolate the UI issue, first verify the source schema directly in a notebook:

df = (
spark.read
.format("binaryFile")
.option("pathGlobFilter", "*.pdf")
.load("/Volumes/<catalog>/<schema>/<volume>/<folder>/")
)
df.printSchema()

The binaryFile reader returns content as BinaryType, alongside path, modificationTime, and length.

If this produces the correct schema while the ingestion UI generates STRING, that helps narrow the problem to the generated table definition or UI configuration.

Could you share the generated SQL and exact type-mismatch error, with sensitive paths removed? Also check whether the destination table already defines content as STRING.