System generated Service Principal -App

[rkhbo3003]

I created an App and creating agent to generate ETL pipeline through prompts. User interaction is with app, however when I do schema check on ADLS it says user does not have access. I cannot see SP in UI as it is system generated, even though I have provided all access to all workspace users, still issue exist.

[balajij8]

@rkhbo3003 

Databricks provisions a dedicated, system generated service principal to act as the App's identity. This service principal evaluates permissions completely independently of any workspace user and hence granting access to all workspace users does not cover the app. Databricks uses this identity to evaluate the app's permissions independently of any user so the app can only access resources explicitly granted to it. While the SP does not appear in the workspace admin users list, you can find its unique ID directly on the Authorization tab of app in the Databricks UI to use in the GRANT statements for providing access.

You can grant the app's service principal Unity Catalog privileges based on how the agent accesses the data to fix the ADLS access issue

• Managed/External Tables - If the agent queries existing external tables registered in Unity Catalog that point to ADLS, the SP requires USE CATALOG on the catalog, USE SCHEMA on the schema and SELECT on the table.
• Direct File Reads - If the agent reads files directly from ADLS, you can assign GRANT READ FILES ON EXTERNAL LOCATION <location_name> TO <app-sp-id> according to the Unity Catalog privileges reference.
  Volumes - If the data access path goes through a Unity Catalog volume, give GRANT READ VOLUME ON VOLUME <volume_name> TO `<app-sp-id>

GRANT READ FILES ON EXTERNAL LOCATION <location_name> TO <app-sp-id>
GRANT READ VOLUME ON VOLUME <volume_name> TO `<app-sp-id>

[rkhbo3003]

I tried that but getting access issues

[balajij8]

You can check the authorization mode the app is using. If the app is configured for user authorization (acting on behalf of the user) rather than app authorization (using the service principal’s identity), the effective permissions come from the interacting user, not the service principal. Error may indicate that the user interacting with the app doesn’t have access to ADLS, even though the service principal does in that case.

You can verify it in the app’s Authorization tab. If it’s using user authorization, the user will need the same Unity Catalog grants. Check if below grants are provided.

GRANT CAN USE ON WAREHOUSE <warehouse_name> TO `<app-sp-id>`;

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<app-sp-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<app-sp-id>`;

GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<app-sp-id>`;

GRANT READ FILES ON EXTERNAL LOCATION <external_location_name> TO `<app-sp-id>`;

GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION <external_location_name> TO `<app-sp-id>`;
GRANT CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<app-sp-id>`;

 Check if the correct SP id is used in all grants

[Satyasai]

Step-by-Step Solution

Step 1: Identify the System Service Principal

Because system-generated SPs are hidden from the standard Workspace Users UI, you must query it using the Databricks CLI or REST API.

Run this command in the Databricks CLI:

Bash Command -

databricks apps get <your-app-name>

Look for the service_principal_id or service_principal_client_id field in the JSON payload output.

Step 2: Grant ADLS Gen2 RBAC Access in Azure

1. Open the Azure Portal and navigate to your ADLS Gen2 Storage Account.
2. Go to Access Control (IAM) $\rightarrow$ Add role assignment.
3. Select Storage Blob Data Contributor (or Storage Blob Data Reader for read-only schema checks).
4. Assign access to the Client ID retrieved in Step 1.

Step 3: Grant Unity Catalog Privileges (If using Unity Catalog)

If your app checks schemas via Unity Catalog External Locations or Volumes, grant permissions directly to the app's Service Principal or service account:

SQL

-- Grant access to the App Service Principal in Databricks

GRANT USE LOCATION ON EXTERNAL LOCATION `<your_external_location_name>` TO `<app-sp-client-id>`;

GRANT READ FILES ON EXTERNAL LOCATION `<your_external_location_name>` TO `<app-sp- client-id>`;