- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday
Hi @ChristianRRL I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tables (repos, issues, PRs). Every command and every step stays the same.
Step 1 — Install & authenticate
Install the community-connector CLI, point it at your workspace profile. Identical for any connector.
Step 2 — Create the connection
community-connector create_connection <CONNECTOR> <conn_name> -o '<options JSON>'
Registers a Unity Catalog connection securely holding the source config/credentials. Only the connector name and the options JSON differ.
What I ran (FHIR):
community-connector create_connection fhir adnan_fhir_conn \
--spec src/databricks/labs/community_connector/sources/fhir/connector_spec.yaml \
--options '{"base_url": "https://hapi.fhir.org/baseR4", "auth_type": "none"}'
Step 3 — Define what to ingest
A small pipeline_spec.json listing source tables → destination catalog/schema. Only the table names differ.
Generic — a pipeline_spec.json listing source tables → destination:
{
"connection_name": "<CONN_NAME>",
"objects": [
{ "table": { "source_table": "<SOURCE_TABLE>",
"destination_catalog": "<CATALOG>",
"destination_schema": "<SCHEMA>",
"destination_table": "<TABLE>" } }
]
}
Step 4 — Create the pipeline (Identical shape for any connector.)
community-connector create_pipeline <CONNECTOR> <pipeline_name> \
--pipeline-spec pipeline_spec.json --catalog <cat> --schema <schema>
What I ran (FHIR) — fhir_pipeline_spec.json:
{
"connection_name": "adnan_fhir_conn",
"objects": [
{ "table": { "source_table": "Patient", "destination_catalog": "uc_adnan", "destination_schema": "fhir_raw", "destination_table": "patient" } },
{ "table": { "source_table": "Condition", "destination_catalog": "uc_adnan", "destination_schema": "fhir_raw", "destination_table": "condition" } },
{ "table": { "source_table": "Encounter", "destination_catalog": "uc_adnan", "destination_schema": "fhir_raw", "destination_table": "encounter" } }
]
}
Step 5 — Run & verify
community-connector run_pipeline <pipeline_name>
community-connector show_pipeline <pipeline_name>
community-connector create_pipeline fhir adnan_fhir_pipeline \
--pipeline-spec fhir_pipeline_spec.json \
--catalog uc_adnan --schema fhir_raw \
--repo-url https://github.com/databrickslabs/lakeflow-community-connectors.git \
--use-workspace-pipeline
Then query the Delta tables in UC. Identical for any connector.