For most engineering teams, Jira is the system of record for project work: issues, sprints, and workflows. And Confluence sits alongside Jira as the knowledge layer, capturing the documentation and decisions that give that work context. Lakeflow Connect makes it easy to unlock insights from both systems, with fully managed connectors that ingest directly into Databricks to enrich your AI agents with trusted enterprise context.
This post covers best practices for setting up the Lakeflow Connect Jira connector and key decisions to make along the way. The Jira connector is currently in Beta and available today through the Lakeflow Connect UI and API. At the end of this blog, we'll show how to pair it with the Lakeflow Connect Confluence connector, now generally available, for a complete Atlassian data pipeline into Databricks.
Step 1: Configure Jira
First, you'll create an OAuth app in the Atlassian developer console and grant it the permissions the connector needs to read from your Jira instance. At the end of this step, you'll have a Client ID and Client Secret that the Databricks Jira connector will use to authenticate. The exact steps vary depending on whether you're on Jira Cloud or Jira Data Center. See Configure Jira for ingestion for more details.
Permissions
When creating the OAuth app, it's important to use an Atlassian account with the right level of access. A standard Jira account is sufficient for most project metadata: boards, sprints, issue types, and priorities. For richer data like project components, fields, and versions, the account needs BROWSE PROJECTS. For user and group data, it needs BROWSE USERS AND GROUPS. ADMINISTER GLOBAL is required for all incrementally updated tables (issues, comments, field values, links, watchers, and worklogs) as well as most admin and security tables. The Jira connector reference page includes a full table of supported tables, their sync type (incremental or full refresh), and the permissions required for each.
Tracking deleted issues
To incrementally ingest deleted issues, the connector reads Jira's audit log, which requires ADMINISTER GLOBAL permissions. On Jira Cloud, a paid plan is also required. Without both, deleted issues will not be reflected in your Databricks issues table.
For comments and worklogs, deletions aren't captured on incremental runs regardless of permissions. A full refresh is the only option.
If you need data from a Jira object that isn't yet covered, reach out to the Databricks Support team to request it.
Step 2: Create a Unity Catalog Connection
Next, you'll create a connection in Unity Catalog using the Client ID, Client Secret, and Jira domain from step 1. A connection is a Unity Catalog object that securely stores your credentials. Once created, the connection can be reused across multiple pipelines without re-entering credentials. For full instructions, see Create a Jira connection. Note: the Jira domain you enter in the Unity Catalog connection must match the site selected in the OAuth app's dropdown in the Atlassian developer console. Double-check that both values point to the same Atlassian site before proceeding to the next step.
Step 3: Create the Pipeline
With the connection in place, you'll create an ingestion pipeline that pulls data from Jira into Delta tables in your Unity Catalog. The pipeline runs on a schedule you define and automatically keeps your tables up to date. You can create the pipeline through the UI under Data Ingestion → Lakeflow Connect, or via the Databricks SDK. Sample code for both approaches is in Ingest data from Jira.
Scoping which Jira projects to sync
By default, the connector pulls from every Jira project the connected account can access. If you only need data from specific projects, you can limit the scope using connector_options.jira_options.include_spaces, which takes a list of project keys. We recommend doing this because it speeds up your initial sync and prevents the pipeline from failing on archived or restricted projects that the account may not have full access to.
The filter applies to the following tables: issues, projects, boards, project_roles, security_schemes, issue_comments, issue_worklogs, issue_watchers, issue_links, issue_field_values, status, version, project_components, project_permissions, project_role_actor, sprints, project_board, and security_level (on-prem only). Tables not in this list are ingested globally regardless of project filter settings.
Managing API quota
Some Jira tables — particularly issue_comments, issue_field_values, and issue_watchers — consume significantly more API quota than others. If you're ingesting a large Jira instance, including these in the same pipeline as your core issue data can slow all your syncs through rate limiting. We recommend creating a separate pipeline for these high-volume tables and scheduling it at a less frequent cadence than your primary pipeline.
Add Confluence to Unify Your Engineering Data for a Complete Picture
Setting up the Confluence connector follows the same three-step pattern: create an OAuth app, create a Unity Catalog connection, and create a pipeline. The connector syncs pages, blog posts, attachments, and space metadata into Delta tables and keeps them up to date incrementally.
With both pipelines running, you can join Jira and Confluence data to unlock several use cases:
- AI assistants: Combine Confluence documentation with Jira issue history to build context-aware AI assistants that can answer questions about both project status and the institutional knowledge behind it.
- Engineering dashboards: Enrich Jira sprint and velocity metrics with data stored in Confluence, including references to the design docs and post-mortems.
- Onboarding: Surface the Confluence spaces most relevant to a new team member based on which Jira projects they are assigned.
For full setup instructions, see the Lakeflow Connect Confluence documentation.
FAQ
What is Lakeflow Connect?
Lakeflow Connect is Databricks' native data ingestion service. It provides managed connectors for popular SaaS applications — including Jira and Confluence — and databases that handle incremental reads, rate limiting, and retries automatically, and land data directly into Unity Catalog as Delta tables.
Do I need to write any code?
No, you can set up the Jira connector entirely using the Lakeflow Connect UI. If you prefer to automate or script your setup, the Databricks SDK is also supported.
Does this work with both Jira Cloud and Jira Data Center?
Yes, both are supported. The OAuth app setup differs slightly between the two — see Configure Jira for ingestion for details.
Do I need admin-level Jira permissions to use the connector?
No, a standard Jira account covers most project metadata. ADMINISTER GLOBAL is required only if you need incrementally updated tables like issues, comments, and field values, or if you want deleted issues tracked.
Is the Jira connector production-ready?
No, the Jira connector is currently in Beta, and therefore does not support production workloads or have SLA support. The Confluence connector is GA (generally available) and is production-ready. Databricks regularly releases previews to allow you to evaluate and provide feedback on features before they are GA. More information on Databricks preview releases is here.
Conclusion
With Jira and Confluence both available in Lakeflow Connect, you can get a full Atlassian data pipeline running in Databricks in a few steps, no custom ETL required. For the Jira connector specifically, the main things to get right upfront are permissions and project scoping.
For full configuration details, see the Lakeflow Connect Jira documentation and the Lakeflow Connect Confluence documentation.
