Lu_Wang_ENB_DBX
Databricks Employee
Databricks Employee

Yes — defining the schema manually for production is okay when you are not using Auto Loader.

With a manually provided schema, you should expect stricter enforcement: Delta will not automatically absorb non-widening type changes like INT -> STRING in append mode.

So the practical recommendation is:

  1. Use a fixed contract at the ingestion boundary if your downstream table is a curated production Delta table.
  2. Handle drift before the final write — either by:
    • normalizing/casting in code, or
    • landing raw data in a staging/bronze table and quarantining bad/type-drifted records for later reconciliation.
  3. If the drift is only a supported widening change, you can enable type widening on the Delta table; otherwise, manual schema alone will not solve the issue.

Summary: manual schema is not a graceful solution for arbitrary source type changes by itself.