Lakeflow Connect for Jira - Questions

[greengil]

So we got the Lakeflow Connect set up with Jira working and so far so good.  But when running the ETL pipeline to inject Jira data into Databricks, there's one table (isssue_with_deletes) fails due to the following error (partially shown).  We have followed the instructions on this page: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/jira-source-setup.

org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = xxxxxxxxxxxxxxxx, runId = xxxxxxxxxxxx] terminated with exception: Job aborted due to stage failure: Task 0 in stage 146.0 failed 4 times, most recent failure: Lost task 0.3 in stage 146.0 (TID 149) (10.0.15.32 executor 0): com.databricks.pipelines.execution.conduit.common.DataConnectorException: [JIRA_ADMIN_PERMISSION_MISSING] Error encountered while calling Jira APIs. Source API type: sourceApi.jira.fetchAuditLogs. Ensure the connecting user has Jira admin permissions for your Jira instance.

Looking at this page: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/jira-reference#supported-jira-source-t...it mentions the user needs global admin permissions in Jira.

When setting up the connection in Catalog, our Databricks admin sets it up with his Databrick admin account.  His  Databricks account has no admin privileges in Jira.  In the setup process, when clicking on 'Sign into Jira" button, we entered the actual Jira admin credentials to log into Jira.  My understanding is, this is a one-time connection establishment between Jira and Databricks.  The actual required permissions are set in the Jira Oauth app scope in the Oauth app.  Reading the above page, it seems like the Databricks admin account used to set up the connection also needs Jira admin for the data injection to work.  Is that the expectation?  If so, why then we need to set the Oauth app scope?  Thanks.

[GaneshI]

Great question — and your confusion is completely understandable. Let me clarify exactly what's happening here.

---

Root Cause

Your diagnosis is correct. Issue deletes are only tracked if the authenticating user (i.e., the Jira account used during the OAuth "Sign into Jira" flow) has global admin permissions on the Jira instance. The issue_with_deletes table specifically relies on the Jira Audit Logs API, and that API is restricted to Jira global admins — it's a hard constraint enforced by Jira itself, not by Databricks.

So the problem is this: when your Databricks admin clicked "Sign into Jira," they authenticated with a Jira account that does not have Jira global admin privileges. Even though the OAuth app was configured correctly (with the right scopes), the actual Jira user identity behind that OAuth token lacks the Jira-side permission to call the audit logs API.

---

To answer your specific question about OAuth scopes

The OAuth app scopes and the authenticating user's Jira permissions are two separate layers — both must be satisfied:

• OAuth scopes define what the application is allowed to request on behalf of a user (e.g., read:audit-log:jira).
• Jira user permissions determine whether the specific authenticated user is actually authorized to access those APIs on the Jira server side.

Think of it like this: the OAuth scope is the key, and the user's Jira admin role is the lock. Even with the right key, if the authenticated user isn't a Jira global admin, Jira's API will reject the audit log calls — which is exactly the JIRA_ADMIN_PERMISSION_MISSING error you're seeing.

---

How to Fix It

Re-do the OAuth connection using a Jira account that has Jira global admin privileges. Specifically:

1. Go back to the connection setup in Databricks Catalog.
2. When you click "Sign into Jira," authenticate with a Jira account that has Jira global admin permissions (not just the Databricks admin account).
3. Once the connection is re-established under a Jira-admin identity, re-run the pipeline.

The OAuth app itself should already have the required scopes including read:audit-log:jira, so you shouldn't need to touch the OAuth app configuration — just re-authenticate with the right Jira user.

---

Alternative if you can't grant Jira admin

If your organization can't grant admin-level scopes or permissions, you can still ingest the tables that don't depend on them — just exclude issue_with_deletes from your pipeline and use the regular issues table instead. You'd lose delete tracking, but the rest of the ingestion will work fine.

[greengil]

@GaneshI Thank you for the detailed explaination!

First, we did "sign into Jira" with a Jira admin user account, though I think there were some issues in the process.  Using the Chrome browser, when that authentication pops up, we enter the Jira admin email address to sign in, but the process doesn't seem to use that account but keep using the Databricks admin account to sign in.  Using the Safari browser, we see the "account icon" on top-right of the authentication screen, and we click on it to log out the Databricks admin account for Jira, and then log in with a Jira admin account.  We are able to sign into Jira.  I guess maybe that doesn't not really sign in with the actual Jira admin account but instead the Databricks admin account itself which doesn't have Jira admin privileges.  So I wonder how to fix this first login issue.

Regarding the 2 layers you mentioned, does that mean Databricks does store the Jira admin credentials and for the subsequent pipelines runs down the road, Databricks needs the Jira admin privileges to do certain things?  If so, we'll need a dedicated "service account" for the process.  To my understand, that initial Jira authentication discussed above is only for a 1-time only to establish the connectivity between Databricks and Jira which need Jira admin privileges.

Finally, you mentioned about excluding the issue_with_deletes, in our pipelines, I specify the 'issues' table to be imported.  How do I tell the pipelines to import Jira issues but not the deleted ones?  Eventually, we'll need deleted issues in Databricks.

Thank you for your help!