Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
Data Ingestion (Lakeflow) Feature Bug:
While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default data type as String, which gives error as Data Type mismatch.
This bug needs to be fixed by Databricks
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago - last edited 2 weeks ago
Reading the PDFs with the binaryFile format returns a fixed schema (path, modificationTime, length, content) with content already typed as BINARY, so there's no column type to set (binary file data source, read_files). You can run this in the SQL editor on a pro or serverless SQL warehouse (requirements) :
CREATE OR REFRESH STREAMING TABLE <catalog>.<schema>.pdf_raw
AS SELECT * FROM STREAM read_files(
'/Volumes/<catalog>/<schema>/<volume>/<folder>/',
format => 'binaryFile',
fileNamePattern => '*.{pdf,PDF}'
);
Add SCHEDULE EVERY 1 hour after the table name if you want scheduled refreshes (standalone streaming tables).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
hI, @kapil_dua., to isolate the UI issue, first verify the source schema directly in a notebook:
df = (
spark.read
.format("binaryFile")
.option("pathGlobFilter", "*.pdf")
.load("/Volumes/<catalog>/<schema>/<volume>/<folder>/")
)
df.printSchema()
The binaryFile reader returns content as BinaryType, alongside path, modificationTime, and length.
If this produces the correct schema while the ingestion UI generates STRING, that helps narrow the problem to the generated table definition or UI configuration.
Could you share the generated SQL and exact type-mismatch error, with sensitive paths removed? Also check whether the destination table already defines content as STRING.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago