Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-06-2026 11:05 AM
- Auto Loader does not change column types on its own. It can add new columns automatically, but won’t widen types of existing columns.
- With your current settings and an initially inferred Quantity INT, if the next CSV has float values, Auto Loader will try to parse them as INT. Those values won’t load as floats; they’ll either become NULL and/or be captured in _rescued_data (depending on the format and parsing rules). They won’t reach your target Quantity as floats.
- Type widening is a Delta Lake table feature (DR 15.4+), not an Auto Loader feature. It can widen INT → DOUBLE/DECIMAL/LONG, but only during writes and when enabled on the target table. You still need the read side to produce data of the widened type (e.g., cast to DOUBLE) before MERGE.
- Auto Loader won’t “round” floats to ints for you. It doesn’t silently coerce 4.5 → 4 or 2.0 → 2. If you explicitly cast in Spark, standard casting rules apply (fraction truncated), but Auto Loader itself avoids implicit lossy conversions for data-quality reasons.
