Hey @Yogasathyandrun , I did some digging and would like to share some thoughts that you hopefully find useful.

You've mapped the boundary here more accurately than most people do, so let me give you a quick reality check on your four sticking points and then take the "is this a wheel" question at the end.

Start with Photon fallback on shared and serverless. This is a product observability boundary, not a gap in your work. Classic all-purpose and jobs compute expose it in the Spark UI DAG, where Photon shows up orange and Spark shows up blue. SQL warehouses and serverless expose it in the Query Profile, where you get "% of task time in Photon" along with operator coloring and the COVERAGE_PHOTON query-performance insight. The catch is that all of this is surfaced in the UI, not made machine-readable. system.query.history has no Photon column, and it only covers warehouse and serverless usage, admin-only at that. Since Spark Connect seals the JVM on shared and serverless, in-process plan inspection is effectively your only programmatic path, and only on classic dedicated compute. For serverless and SQL, I'd triage from the Query Profile or COVERAGE_PHOTON rather than promise per-statement detection.

On fleet rollout, the least intrusive approach is to pin spark.extraListeners through a cluster policy, ship the JAR from a UC volume that sits on the UC allowlist (you'll need MANAGE ALLOWLIST), and keep com.databricks...DBCEventLoggingListener in the list. OpenLineage uses this same pattern, so you're on well-worn ground. Skip global init scripts. They don't run on standard or serverless, and DBFS init scripts are end-of-life. There's a split worth keeping in mind: your QueryExecutionListener is dedicated-only, since there's no QueryExecution under Spark Connect. A plain SparkListener may travel to shared compute, but test it on your DBR before you count on it. Serverless honors neither.

For the observation window, a hybrid approach works best. The public Databricks system-tables and LakeFlow dashboard uses a rolling 30-day window and reports peak alongside average, which is a sensible default since it spans a full monthly cycle. Gate any downsizing on peak or p95 to p99, not on the mean, and keep hard vetoes for spill, swap, and sustained driver memory pressure. Optimized autoscaling already scales down conservatively, so tuning min and max workers often beats re-sizing a fixed cluster.

Billing attribution has a few traps. Handle the open price window (price_end_time IS NULL OR usage_end_time < price_end_time) or your current usage will drop out. Remember that list_prices is list price, not your contract rate. usage_metadata.cluster_id is null for SQL warehouses and all serverless, so coalesce it with warehouse_id, job_id, and the rest, then branch on billing_origin_product. Serverless emits multiple records, so sum them. Billing is global while the compute tables are regional. And product_features.is_photon is an enablement flag, not a fallback signal, so don't conflate the two.

Now the question you really care about: are you reinventing the wheel? For generic cost and right-sizing, mostly yes. The system-tables usage dashboards, Overwatch (much of which is now reachable through system tables), and the third-party FinOps tools all cover that ground. What none of them cover is Photon fallback and Photon ROI. There's no first-class fallback metric to consume, so everyone skips it. That's the real opening, and it's the strongest part of your prototype. I'd lead with Photon ROI and treat cost and right-sizing as the expected baseline. Just design around the one hard constraint: that signal is only reachable programmatically on classic dedicated compute today.

Hope this helps you focus the build.

Cheers, Lou.