Databricks Community

dbernstein_tp · 7 hours ago

Hi Everyone, This is a follow up to a previous question I posted. We have a need for a number of apps that have a spreadsheet-like interface and read/write to our lakehouse. A common spec that comes up in these discussions is the need for managers to view who made what changes to the data and when. As in "if I hover over a cell can I see the last edit to the cell? When was it made, who made it, and what the change was?"

Is this possible for a dbx app, say using Streamlit and perhaps a Lakebase backend? Or is it just not doable currently?

balajij8 · 7 hours ago

Hi dbernstein_tp,

Yes. Its possible on Databricks Apps using Streamlit or others. The challenge comes down to balancing write latency for an interactive grid against how you store and query the cell-level history.

For the frontend, st.data_editor gives you the native spreadsheet UI with inline editing and change detection. Streamlit’s st.column_config allows basic tooltips, but for hover-based audit metadata (showing prior values, timestam, and user), you can use popovers, conditional cell styling, or light HTML/CSS injection to display the cell's history when selected or hovered. To capture user identity cleanly, extract the authenticated user context from the app's request headers or OAuth token context via the Databricks SDK rather than relying on client-side inputs.

For storage backend and audit strategy, you can use Lakebase or Databricks SQL based on speed needs.

Lakebase Autoscaling

If the users expect snappy, sub-second cell saves, Lakebase (Postgres-backed OLTP) is the better fit. You get < 50ms write latencies and can offload the audit trail entirely to Postgres triggers on the primary table. Every UPDATE or INSERT automatically writes the table_name, column_name, row_id, old_value, new_value, changed_by, and changed_at into a dedicated audit_log table transactionally. When the Streamlit grid loads, you join or fetch against this log to populate hover context.

Databricks SQL

If you want to keep everything strictly inside Delta tables, enable Change Data Feed (CDF) on your target table. You can track cell-level modifications and query history via table_changes(). The tradeoff here is write latency as Delta commits typically take a few seconds per write transaction, which can feel slow if managers are editing row-by-row in a real-time grid. It works well if edits are flushed in batch, but for real-time cell feedback, it requires handling asynchronous writes in the app layer.

If you need the native spreadsheet feel where edits write back instantly and populating the audit tooltip shouldn't bottleneck the UI, Lakebase gives you the lowest friction setup.

szymon_dybczak · 7 hours ago

Hi @dbernstein_tp ,

Yes - this is doable in a Databricks App, but I don’t think there is currently a native “hover over any cell and Databricks automatically tells me who changed it, when, and from what value” capability. You would build that behavior into the app.

A Streamlit Databricks App can already read a Unity Catalog table, allow edits, and write them back. Databricks has an official example of exactly that pattern.

For the audit requirement, there are a couple of approaches:

1. Lakebase as the transactional backend - probably the cleanest approach for this type of app.

Lakebase now also has Change Data Feed, which captures every INSERT, UPDATE, and DELETE, including the before/after row state, transaction information, and timestamp, and writes the history into an immutable Delta table in Unity Catalog. Databricks explicitly lists audit history as a use case.

You would still need to capture the application user identity with each edit. That is important because an App using normal app authorization writes as its service principal, so otherwise every change can look like it came from the application rather than Alice/Bob/etc.

Databricks Apps now also support user authorization, where the app acts using the identity of the user interacting with it.

Then when the user hovers/clicks a cell, the UI looks up the most recent audit entry for that row + column. You could also expose the full change history.

2. Write directly to Delta.

Delta Change Data Feed can give you row-level "update_preimage" and "update_postimage" records plus commit version and timestamp. Delta table history can also tell you which identity performed each table write.

dbernstein_tp · 6 hours ago

Thank you both for those very quick and excellent responses. The hovering thing is just an example of the kind of question I get, probably not an exact spec. I would be happy if the audit trail can be displayed somehow in the app.