topic Re: Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type in Data Engineering

Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

kapil_dua — Tue, 03 Nov 2026 10:38:00 GMT

Data Ingestion (Lakeflow) Feature Bug:

While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default data type as String, which gives error as Data Type mismatch.

This bug needs to be fixed by Databricks

Re: Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

AbhilashNagilla — Wed, 04 Nov 2026 04:54:26 GMT

Reading the PDFs with the binaryFile format returns a fixed schema (path, modificationTime, length, content) with content already typed as BINARY, so there's no column type to set (binary file data source, read_files). You can run this in the SQL editor on a pro or serverless SQL warehouse (requirements) :

CREATE OR REFRESH STREAMING TABLE <catalog>.<schema>.pdf_raw
AS SELECT * FROM STREAM read_files(
  '/Volumes/<catalog>/<schema>/<volume>/<folder>/',
  format => 'binaryFile',
  fileNamePattern => '*.{pdf,PDF}'
);

Add SCHEDULE EVERY 1 hour after the table name if you want scheduled refreshes (standalone streaming tables).

Re: Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

Jen_Evangelista — Fri, 06 Nov 2026 04:19:14 GMT

hI, @kapil_dua., to isolate the UI issue, first verify the source schema directly in a notebook:

df = (
spark.read
.format("binaryFile")
.option("pathGlobFilter", "*.pdf")
.load("/Volumes/<catalog>/<schema>/<volume>/<folder>/")
)
df.printSchema()

The binaryFile reader returns content as BinaryType, alongside path, modificationTime, and length.

If this produces the correct schema while the ingestion UI generates STRING, that helps narrow the problem to the generated table definition or UI configuration.

Could you share the generated SQL and exact type-mismatch error, with sensitive paths removed? Also check whether the destination table already defines content as STRING.

Re: Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

Jen_Evangelista — Fri, 06 Nov 2026 04:20:30 GMT

+
source
https://docs.databricks.com/aws/en/query/formats/binary

Re: Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

saurabh18cs — Fri, 06 Nov 2026 15:35:34 GMT

Hi @kapil_dua This works for me so what are you doing differently?