savlahanish27
Databricks Partner

Hi Ashwin,

Really clean write-up - the schema override demo with the actual DESCRIBE DETAIL output is what makes it click. Most posts on this topic stop at "here's the theory," this one actually shows it happening.

Something similar came up on a SAP HANA to Databricks migration I worked on, with a slight twist on the landing zone side.

We had two paths feeding into Bronze - Oracle GoldenGate pushing CDC events through Kafka into Structured Streaming, and a separate JDBC job handling the historical backfill. Both needed to write to a fixed path before Databricks touched anything. GoldenGate especially - it was configured to drop files at a specific ADLS location and that path had to just exist, stable, regardless of what we were doing inside Unity Catalog. So, the landing zone ahead of Bronze ended up external, basically your "fixed path, register-in-place" case exactly.

Once Structured Streaming picked those files up and wrote them into Bronze, everything from there on was managed. Honestly that was one of the better decisions - predictive optimization and auto VACUUM just ran on their own, one less job we had to maintain. By Gold, everything was managed, partitioned by company code and fiscal year, Z-ordered on the columns that got hit hardest in joins.

So, for us it landed as: external just for that one pre-bronze hop, managed for everything after. Curious if your banking customer's CDC setup let them skip that step entirely and write straight into a managed volume, or if they hit the same fixed-path constraint we did.