SFDataEng
Contributor

Agreement

Yes, a feature to support nested datasets in Databricks AI/BI dashboards would seem convenient, e.g. as in both examples in the OP's screenshot.

Specific need

A use case that comes to mind would be to avoid the maintenance of parallel code across dashboard SQL datasets that contains parameters driven by dashboard widgets/filters, which might ideally be defined and located once in an "inner" dataset.

Why it's needed

The maintenance inefficiencies that arise in the absence of such a feature can include, for example,

  • keeping a common CTE in sync across datasets, and also
  • linking each and every dashboard filter to each and every instance of that parameter's usage across all datasets.

The pain

This parallel definition has potential to multiply out rather unsuitably for a dashboard with many (common) dashboard-wide filter widgets/parameters and many charts, each having its own custom dataset meticulously defining metrics, in terms of dashboard construction, maintenance, and reliability.  Nonetheless, the "hard/inefficient-mantenace way" seems possible as a brute force workaround to the lack of "nested dataset" support.

Scope and non-alternatives

Obviously, this problem/opportunity applies mainly to datasets that are "dynamic" and not suitable for being fully pre-materialized; pipelining a table or materialized view does not solve this problem.  To be clear, regular views also are not suitable because views cannot contain parameters.  (If view supported `:parameters` to be filled in at run time, that could be an option).

Example in another dashboard tool

In contrast, and perhaps for inspiration, dashboarding in Apache Superset supports a built-in dataset() macro for n-level nested dataset usage, which works quite nicely.  This is basically the ask in the OP -- to be able to query/reference another dataset in the same query.

Possible (non-applicable?) alternative approach in Databricks

A potential alternative that appears to exist within the Databricks ecosystem for further exploration might be the Unity catalog metric views.  (Still, the possibility of nested Dashboard datasets, to make use of "common parameters" in the core/central/sub-datasets, would be a most welcome feature addition, since some use cases might not rise to the occasion of adopting a whole "metrics framework".)

More critically, as an apparent blocker, however, the notice


"Datasets with parameters cannot be used to export to metric veiws"

gets raised in the Databricks AI/BI dashboard interface when attempting to create a metric view from a Dashboard dataset that contains parameters.  Therefore, it also seems possible that Metric Views in Unity Catalog may not in fact represent an applicable solution at all, for the scenario in which the dataset to be nested contains parameters (which, in this case, is a primary reason for attempting to make such a reusable view in the first place).