Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 hours ago
Yes — defining the schema manually for production is okay when you are not using Auto Loader.
With a manually provided schema, you should expect stricter enforcement: Delta will not automatically absorb non-widening type changes like INT -> STRING in append mode.
So the practical recommendation is:
- Use a fixed contract at the ingestion boundary if your downstream table is a curated production Delta table.
- Handle drift before the final write — either by:
- normalizing/casting in code, or
- landing raw data in a staging/bronze table and quarantining bad/type-drifted records for later reconciliation.
- If the drift is only a supported widening change, you can enable type widening on the Delta table; otherwise, manual schema alone will not solve the issue.
Summary: manual schema is not a graceful solution for arbitrary source type changes by itself.