Unmasking the Connected Consumer

[KeerthiItigimat]

Behind every clean “Golden Record” is a messy web of shared emails, phones, and legacy IDs. Here’s how we turned that web into an interactive, cluster-aware graph a data steward can actually work with — built on Streamlit, NetworkX, and a Databricks SQL Warehouse.

In the world of Master Data Management, the “Single View of the Customer” is the holy grail. But behind every clean Golden Record lies a messy web of raw data: five different email addresses, shared phone numbers, and legacy system IDs that may or may not belong to the same person.

Identifying these links is identity resolution. Visualizing them, however, is where the real insights happen — and where we built the Entity Linkage Explorer: a Streamlit app that lets data stewards traverse identity clusters, isolate specific data “islands,” and filter out noisy attributes in real time, all against a live Databricks SQL Warehouse.

Figure 1. The Entity Linkage Explorer running as a Databricks App — the sidebar drives what the graph shows; the canvas renders only the clusters the steward has toggled on.

01
From tables to topology

Traditional data quality tools give you a table. But tables are terrible at showing transitive relationships. If Person A shares an email with Person B, and Person B shares a phone number with Person C, a table makes it hard to see that all three might be the same household.

Tables show rows. They don’t show the shape of a relationship.

The core problem

We needed a tool that could:

• Query scale. Pull deep lineage from a Databricks SQL Warehouse without dragging the whole table across the wire.
• Identify clusters. Automatically group connected individuals into “islands.”
• Handle noise. Dynamically exclude junk data — test@test.com, 123 Main St — that causes false links.
• Perform. Render hundreds of nodes without crashing the browser.

02
A four-tier, graph-first architecture

The app separates security, data, logic, and rendering into their own tiers — each one only doing the job it’s good at.

Figure 2. The app captures the user’s OBO token to establish a secure session, pulls only the columns needed for linking, builds a logical graph with NetworkX, then hands it to PyVis to render.

To keep the data tier fast, we avoid SELECT * and pull only the attributes necessary for linking. Before drawing anything, we build a logical graph in NetworkX: every person and every attribute — an email, a phone number, an address — becomes a node, and a shared attribute becomes an edge between the people who share it. Running Connected Components over that graph instantly identifies every isolated cluster in the dataset, no matter how many hops apart the shared attribute is.

03
Cluster filtering tames the hairball

One of the biggest pain points in graph visualization is the “hairball effect” — too many nodes create an unreadable mess. So the app never renders anything until the user has chosen what to look at.

Connected components are computed once, then listed in the sidebar as named, countable clusters:

Users toggle clusters on or off, and only the visible nodes are ever handed to PyVis. If a specific email address is stitching together two families that shouldn’t be linked, the steward can see exactly which cluster it sits in and reach for the Unlink / Exclude feature to remove it — without touching anyone else’s view.

Figure 3. Excluding one bad phone number splits a falsely merged household into two correct clusters — the exact “What-If” check a data steward runs before trusting a linkage.

04
Engineering for performance

Thousands of identity links can turn naive Python loops into the bottleneck. Three specific optimizations keep the UI snappy.

itertuples() instead of iterrows()

iterrows() is notoriously slow because Pandas builds a fresh Series for every row. Switching to itertuples() improved graph-building speed by nearly 50x, processing thousands of links in milliseconds.

Two-layer caching

Streamlit’s @ST.cache_data is split into two layers: a data cache for the raw SQL results, and a graph cache for the NetworkX object itself. Changing a purely visual setting — background color, node size — redraws the existing graph instead of re-querying Databricks or reprocessing the DataFrame.

Physics stabilization

Graph simulations are CPU-intensive. Configuring PyVis’s forceAtlas2Based solver with a high stabilization threshold lets the layout settle before render, instead of animating a bouncing simulation that freezes the browser on large loads.

pyvis network options

net.set_options("""
{
  "physics": {
    "solver": "forceAtlas2Based",
    "stabilization": {"iterations": 150}
  }
}
""")

05
Handling data quality on the fly

Data is rarely perfect. A generic placeholder like NA or a shared “house” phone number will link thousands of unrelated people together if you let it. The Explorer includes a Dynamic Exclusion engine: type any value into a “Values to Remove” box, and the app instantly:

• Drops those nodes from the NetworkX graph.
• Recalculates the clusters.
• Redraws the now-cleaned visualization.

This is the point of the whole tool: entity linkage isn’t just an algorithm, it’s human-in-the-loop verification. NetworkX supplies the analytical power; Streamlit supplies the rapid UI. Together they turn a static lineage report into something relationships can be explored, cleaned, and understood at a glance.

Better data quality. Fewer duplicate customers. A much clearer picture of who
your consumers actually are.