DLT SQL schema definition
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2024 09:24 PM - edited 06-25-2024 09:28 PM
Hi All,
While defining a schema in creating a table using Autoloader and DLT using SQL, I am getting schema mismatch error between the defined schema and inferred schema.
Is there a known limitation or am I missing something here?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2024 11:24 PM
@Sudheer_DB
In your schema there's a column named rescue_data, while the default autoloader column name for faulty data is _rescued_data
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-26-2024 12:49 PM
Hi @daniel_sahal ,
Thank you for your response. The whole idea is to define my own column names. Shouldn't I rename the rescued_data column?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-26-2024 11:26 PM
@Sudheer_DB
You can specify your own _rescued_data column name by setting up rescuedDataColumn option.
https://docs.databricks.com/en/ingestion/auto-loader/schema.html#what-is-the-rescued-data-column