Unable to access all schemas in system catalog

[kartikaybagla]

https://docs.databricks.com/en/administration-guide/system-tables/index.html

Based on this documentation, there should be a bunch of schemas in the system catalog and when trying to list them all, I can only see the INFORMATION_SCHEMA in the system catalog. I am logged in as account admin and my workspace is Unity Catalog governed.

How do I enable access to the billing, compute and access schemas?

[Kaniz]

Hi @kartikaybagla , Thank you for sharing the link to the Databricks documentation on system tables. 

Let’s dive into how you can enable access to the specific schemas you’re interested in.

System Tables Overview:

• System tables in Databricks serve as an analytical store for your account’s operational data. They allow historical observability across your account.
• These tables are useful for tracking various aspects such as audit logs, billable usage, table lineage, and more.

Available System Tables: Currently, Databricks hosts system tables for several areas, including:

• Audit logs: Records all audit events across your Databricks account.
• Table lineage: Tracks read or write events on Unity Catalog tables or paths.
• Column lineage: Records read or write events on Unity Catalog columns (excluding events without a source).
• Billable usage: Contains records for all billable usage across your account, aggregated hourly.
• Pricing: Maintains a historical log of SKU pricing changes.
• Clusters: Provides a full history of cluster configurations over time.
• Node types: Captures available node types and their hardware information.
• Marketplace listing access: Includes consumer info for completed request data or get data events on your listings.
• Predictive optimization: Tracks operation history related to predictive optimization features.

Enabling System Tables:

• Since system tables are governed by Unity Catalog, you need at least one Unity Catalog-enabled workspace in your account.
• System tables include data from all workspaces in your account but can only be accessed from a Unity Catalog-enabled workspace.
• Account admins can enable system tables using the Unity Catalog REST API.
• If you’re interested in using any of these tables, consider reaching out to your Databricks account team.

Billing System Table:

• The billing system table (system.billing.usage) contains data for all workspaces in your account, regardless of the region they’re deployed in.
• While system tables can only be accessed through a Unity Catalog workspace, they also include operational data for non-Unity Catalog workspaces in your account.

[kartikaybagla]

Hi @Kaniz, thank you for replying. I tried enabling system tables using the Unity Catalog REST API but it only shows the Information Schema available and doesn't show any details for all the other schemas. I used an account admin credentials for doing this. How do I fix this issue?

[kartikaybagla]

More specifically, I get this error when using the REST API for the Unity Catalog:

Error enabling the schema `billing`: INVALID_PARAMETER_VALUE | Description: EnableSystemSchema billing is not a system-schema that is available in us-west1

 I get the same error for another workspace in ap-southeast1 region.

[Kaniz]

Hi @kartikaybagla , Thank you for reaching out! 

It seems you’re encountering an issue with the Unity Catalog REST API while trying to enable system tables.

Let’s troubleshoot this together.

Catalogs and Schemas:

• The Unity Catalog organizes data assets into catalogs and schemas. Catalogs are the top-level containers, and schemas reside within catalogs.
• Users can see all catalogs on which they have been assigned the USE_CATALOG data permission.
• To access or list a table or view in a schema, users must also have SELECT permission on the table o....

System Schemas:

• The Unity Catalog includes system schemas like system.operational_data and system.lineage.
• These system schemas might contain empty tables or metadata related to operational aspects of the platform.
• However, not all system schemas are accessible via the REST API by default.

Enabling System Schemas:

• When new schemas are released, an account admin needs to explicitly enable the specific schema for it to be accessible.
• You mentioned encountering an error related to the billing schema. Let’s address that:

Troubleshooting Steps:

• Ensure that you are using the correct schema name (billing) and workspace region (us-west1).
• Verify that the schema you are trying to enable is indeed a system-schema supported by the Unity Catalog.
• Double-check your account admin credentials and permissions.
• If the issue persists, consider a support ticket with us.

Additional Considerations:

• If you encounter the same error in another workspace (e.g., ap-southeast1), follow similar troubleshooting steps for that region.
• Keep an eye on any updates or announcements related to system schemas in the Unity Catalog.

Remember that the Unity Catalog provides centralized data governance across all workspaces in a Databricks account. If you continue to face issues, LMK.

[SanketMehta]

Hi Team, If you have any particular solution for this, then please share as I am also unable to access any system tables I enabled in 2023 August now!!