Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2026 08:16 AM
1. No, Auto Loader does not provide an option to automatically widen column types. Its schema evolution modes are:
addNewColumns → Adds new nullable columns.
rescue → Captures unexpected fields in _rescued_data.
failOnNewColumns → Stops the stream on schema drift.
Type widening is a Delta Lake feature, not an Auto Loader feature. You can enable it on the target table with
ALTER TABLE <table_name> SET TBLPROPERTIES ('delta.enableTypeWidening' = 'true');
2. Yes, Delta Live Tables (DLT) uses Auto Loader under the hood for streaming ingestion. The default behavior is the same:
- Adds new columns when detected.
- Does not change existing column types automatically. For type widening, you rely on Delta Lake’s schema evolution and type widening features in the Silver/Gold layers. DLT does not override Auto Loader’s limitations.
3. Suggested approach for unstable upstream schemas 100s of columns
When explicit schemas or hints are impractical, consider these strategies:
- Use schema-on-read with flexible types
- For CSV/JSON, Auto Loader often infers all columns as STRING if inferColumnTypes is off. You can then cast selectively downstream.
- Alternatively, set cloudFiles.inferColumnTypes=true but combine with schemaHints for critical columns
- Leverage _rescued_data for audit and repair.Keep _rescued_data in Bronze for anomaly detection.
- Apply transformations in Silver to reconcile rescued values when needed.
- Cast numeric columns to DOUBLE or DECIMAL in the Bronze layer.
- Enable column mapping on the Delta table ('delta.columnMapping.mode' = 'name') to handle column renames and drops as metadata-only changes without data rewrites, which improves performance and resilience to structural changes.
