If you are early in a Hadoop to Databricks migration, the artifact worth building before any pilot is a function-by-function control map. For each capability the Hadoop estate currently provides, record three things: the control objective that must still be met after Hadoop is gone, how Databricks will meet it, and what evidence will prove it. A product comparison table gets you started and then stops being useful, because it can leave identity, recovery, scheduling, cost allocation, and data lifecycle without an owner.

This applies to teams running a production HDFS and Hive estate with scheduled Spark or MapReduce workloads, Kerberos-based authentication, and a decommissioning target. It assumes you have workspace and Unity Catalog design authority, or access to whoever does.

Why the component-swap table runs out

A production Hadoop environment provides storage, table metadata and grants, execution, resource arbitration, orchestration, authentication and authorization, monitoring, and a body of operational habit. Those functions have consumers, and the register of consumers is usually incomplete on day one.

Two problems show up when a migration is planned from a swap table alone.

The first is that some Hadoop functions have no single destination. YARN handles isolation, priority, quotas, and chargeback. On Databricks those responsibilities distribute across compute policies, job design, budgets, and operating rules that people have to agree to follow. Nothing in the platform enforces the old queue hierarchy for you, so if the queue structure encoded a chargeback agreement between teams, that agreement needs rebuilding somewhere explicit.

The second is that a mapping row can look satisfied while the objective it existed to serve is unmet. Registering every Hive table in Unity Catalog produces a catalog that looks complete. It can still fail at query time if file locations, client behavior, or effective permissions differ from the source. Completeness of objects and correctness of access are separate claims requiring separate tests.

The three-column map

Populate one row per Hadoop function. The verification column is the one that changes behavior, because it converts an architecture opinion into a task somebody has to complete and sign.

The question each row answers is which control objective must still be met, by whom, and how it will be tested. That phrasing keeps the conversation away from feature equivalence arguments that cannot be settled.

Populating the map from a source-state register

The map is only as good as the inventory behind it. Three record types feed it.

Data record

  hdfs_path, format, partitioning, file_count, volume,

  change_rate, retention, owner, consumers[], mutable

Hive record

  database, table_or_view, location, serde, functions[],

  properties, grants[], owner, metastore_version

Workload record

  repository, language, runtime, libraries[], yarn_queue,

  schedule, retries, inputs[], outputs[], checkpoints,

  secrets[], consumers[], sla, recovery_objectives, approver

These are register field lists rather than executable definitions. Adapt the names to whatever inventory store you already run.

Populate them from several sources, because each one is incomplete in a different way: HDFS listings and fsck reports, metastore queries, scheduler exports, Spark history logs, YARN history, network flow data, audit logs, code search, and interviews with consuming teams. The dependencies that break a cutover tend to sit outside the platform boundary. Hard-coded hdfs:// paths in application code, JDBC clients, shell scripts, unmanaged JARs, keytabs, files on edge nodes, and teams reading an output without owning the job that writes it.

One convention is worth agreeing in writing before discovery starts. A dataset or job with no identified owner is an open risk item, not evidence that the asset is unused.

Boundaries to settle before the pilot

Several decisions constrain every row in the map, and revisiting them mid-program is expensive.

On the data side: storage locations, regions, encryption keys, managed against external tables, catalog and schema design, environment isolation, ownership, and retention. Map databases to catalogs and schemas according to ownership, environment, region, and sharing boundaries rather than copying the existing Hive name structure, since the old names usually encode team history rather than a governance model.

On compute: runtime baselines, the serverless and classic boundary, access modes, autoscaling, dependency installation, and how exceptions get approved.

On orchestration: which system owns retries, backfills, calendars, dependency state, and escalation. Splitting that ownership between a retained enterprise scheduler and Databricks Workflows is workable when the split is explicit about which side holds dependency state.

On networking: private connectivity, DNS, egress, cloud endpoints, on-premises routes, and transfer throughput. Throughput belongs in this list because it sets the shape of the data migration. Where a full recopy will not fit the cutover window, you need bulk transfer followed by controlled incremental synchronization, and the resulting lag has to be measured against the cutover objective rather than estimated.

Operational rows people forget

Two functions in the Hadoop estate are provided by people and process rather than by software, and they need rows of their own.

Observability is the first. Job state, Spark metrics, data-quality results, lineage, audit events, telemetry, and billing all end up in different places on the target platform. Recovery planning should separate rebuilding compute, restoring metadata, replaying pipelines, recovering data, and meeting the business objective, because those have different owners and different durations.

Cost attribution is the second. Where YARN queues carried an implicit chargeback model, tagging discipline and compute policies now carry it. Unrestricted compute, weak tagging, idle resources, repeated backfills, and inefficient file layouts all raise consumption, and the first three are only visible if tags were designed before the landing zone was built rather than added afterward.

Limitations and tradeoffs

The medallion pattern of bronze, silver, and gold layers earns its place when each layer represents a controlled quality transition. Read as an instruction to keep three copies of every dataset, it adds storage and reconciliation work without adding assurance.

Rehosting is not automatically the cheapest disposition. For a job whose code barely changes, data movement, permission reproduction, and testing can dominate the effort. Refactoring carries more engineering uncertainty and can reduce steady-state consumption or operational load, depending on what the original job was doing inefficiently.

Databricks Labs UCX can accelerate assessment and parts of a workspace-local Hive Metastore upgrade. Treat its output as input to your own review, testing, and approval rather than as a completed migration. Confirm its current scope against Databricks documentation, since tooling of this kind changes between releases.

Compatibility findings are version sensitive. Comparing source Spark, Scala, Java, Python, Hive, and Hadoop client versions against the selected Databricks Runtime is a prerequisite for any effort estimate, and so is checking SQL semantics, data types, error behavior, dependencies, streaming behavior, access modes, and serverless constraints as they stand at the time you plan.

There is no defensible generic duration for this work. Estimate from the register and the pilot, then reforecast after each wave.

This article adapts material from Hadoop to Databricks Migration Steps, Risks, and Costs, previously published on the Arbisoft blog. That version covers the seven-phase execution sequence, the gate packs required at each phase, the four-layer cost model, and the risk register in more detail than this piece, which concentrates on the capability map and its verification.

For those who have retired a YARN-based estate: where did the old queue quotas and chargeback agreements end up on Databricks, in compute policies, in budgets and tag-based reporting, or in a written operating agreement between teams? I am interested in which of those held up once concurrency rose.