cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

kapil_dua
Databricks Partner

yesterday

Data Ingestion (Lakeflow) Feature Bug:

While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default data type as String, which gives error as Data Type mismatch.

This bug needs to be fixed by Databricks

Labels:
0 Kudos
1 REPLY 1

AbhilashNagilla
Databricks Employee
Databricks Employee

yesterday - last edited yesterday

Reading the PDFs with the binaryFile format returns a fixed schema (path, modificationTime, length, content) with content already typed as BINARY, so there's no column type to set (binary file data source, read_files). You can run this in the SQL editor on a pro or serverless SQL warehouse (requirements) :

CREATE OR REFRESH STREAMING TABLE <catalog>.<schema>.pdf_raw
AS SELECT * FROM STREAM read_files(
  '/Volumes/<catalog>/<schema>/<volume>/<folder>/',
  format => 'binaryFile',
  fileNamePattern => '*.{pdf,PDF}'
);

Add SCHEDULE EVERY 1 hour after the table name if you want scheduled refreshes (standalone streaming tables).

0 Kudos
Announcements