Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

[kapil_dua]

Data Ingestion (Lakeflow) Feature Bug:

While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default data type as String, which gives error as Data Type mismatch.

This bug needs to be fixed by Databricks

[AbhilashNagilla]

Reading the PDFs with the binaryFile format returns a fixed schema (path, modificationTime, length, content) with content already typed as BINARY, so there's no column type to set (binary file data source, read_files). You can run this in the SQL editor on a pro or serverless SQL warehouse (requirements) :

CREATE OR REFRESH STREAMING TABLE <catalog>.<schema>.pdf_raw
AS SELECT * FROM STREAM read_files(
  '/Volumes/<catalog>/<schema>/<volume>/<folder>/',
  format => 'binaryFile',
  fileNamePattern => '*.{pdf,PDF}'
);

Add SCHEDULE EVERY 1 hour after the table name if you want scheduled refreshes (standalone streaming tables).