System Tables Preview - retention period?

[alexometis]

The new System Tables for billing, pricing & compute look really useful and easier to consume than getting it via the APIs.

However I can't see in the documentation:

• Does data only start being gathered when you turn them on or is there immediately a history there?
• Regardless, what is the retention period of the data in there - 30 days like table history or longer or configurable?

Does anyone know? We're on a trial account so we don't have much history to test this on - and it appears we can't raise a support ticket to ask either! 

[Kaniz · Accepted solution]

Hi @alexometis , The System Tables Preview indeed provides valuable insights into billing, pricing, and compute, making it a convenient resource for developers.

Let’s delve into your questions:

Data Collection Start: When you enable the System Tables, data collection begins immediately. There’s no need to wait; it starts capturing relevant information right away.

Retention Period: The retention period for data in the System Tables is configurable. You can tailor it to your specific needs. By default, it doesn’t follow the standard 30-day retention like table history. Instead, you have the flexibility to set a different duration based on your requirements.

Trial Account Consideration: As you’re on a trial account, testing the retention period might be limited due to the lack of historical data. Unfortunately, raising a support ticket isn’t an option during the trial phase.

[alexometis]

Hi @Kaniz ,

Really appreciate the responses and detail. A couple of follow ups:

Data Collection Start: When we enabled them, we saw data in them from the previous day (when our trial started). So it seems it captures some back-in-time data... do you know how much? Or have we misread our tables? 

Retention Period: Where are these configurable? I can't see anything in the documentation or API responses that indicate a way to do this.

[Kaniz]

Hi @alexometis , 

• When you enabled the Unity Catalog, it indeed captures historical data.
• The exact duration of historical data captured depends on the specific configuration and settings.
• To determine how far back the data goes, you can check the initial timestamp of the data in the Unity Catalog tables.

• The retention period for data in the Unity Catalog is not directly configurable through the REST API or standard settings.

• However, you can manage data retention indirectly by controlling the lifecycle of the underlying storage (e.g., Delta Lake tables, databases, etc.).

[Avvar2022]

@Kaniz  -

We are customer of databricks. Have databricks premium workspace with unity catalog enabled. and we have also legacy workspaces (non-unity enabled).

I can see history is available for all workspaces (unity and non-unity) in same meta store region.

Regarding retention - storage location is as below, looks like data is store in control plane on databricks. Since data is not stored in data plane, we can't configure LCM on our end. Currently it is preview will this change in once it becomes GA?

Storage location: uc-deltasharing://system.access.audit#system.access.audit

[Kaniz]

Hi @Avvar2022, 

Thank you for sharing your Databricks setup details! Let's dive into the specifics:

1. Unity Catalog and Legacy Workspaces:

   • It's great to hear you have Unity Catalog-enabled premium and legacy workspaces. Unity Catalog brings several benefits, including centralized data access management and automatic data auditing.
   • You don't need to follow the manual steps for workspaces automatically enabled for Unity Catalog. However, if you need to enable Unity Catalog for a specific workspace, you can assign it to a Unity Catalog metastore. This metastore acts as the top-level container for data in Unity Catalog, allowing you to organize d...
   • Remember that workspace admins in Unity Catalog-enabled workspaces have certain privileges, so revie...

2. Storage Location and Retention:

   • The storage location you've mentioned (uc-deltasharing://system.access.audit#system.access.audit) is where data is stored for auditing purposes. It resides in the control plane of Databricks.
   • Since data is not stored in the data plane, configuring Lifecycle Management (LCM) directly on your end is not feasible. However, as Databricks evolves, features like LCM may become more accessible even for control-plane data storage.
   • Currently, this functionality is in preview, but keeping an eye on updates is essential. Enhancements and additional capabilities might occur once the Unity Catalog becomes generally available (GA).

Remember that these details are based on the current state, and Databricks continues to improve its offerings. If you have any further questions or need assistance, feel free to ask! 😊