- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Wednesday
Hi,
You can ingest JSON data from an external REST API into Databricks Lakebase (PostgreSQL) using a Databricks notebook.
I would suggest the following approach:
Extract data: Use Python's request library to connect to the external API and retrieve the JSON response. Configure the required authentication (API key, OAuth, etc.).
Transform data: Parse the JSON response and map the fields to your target Lakebase table schema.
Load into Lakebase: You can use the Lakebase Data API, which supports inserting records through HTTP POST requests, or connect directly using a PostgreSQL driver such as psycopg.
Automate ingestion: Schedule the notebook using Databricks Workflows. For larger datasets, consider pagination, batching, incremental loads, and error handling.
Suggested flow:
External REST API → Databricks Notebook (Python) → JSON Processing → Lakebase PostgreSQL Table
If you are using Lakebase Autoscaling, the Data API is a convenient option. Make sure it is enabled and the authenticated identity has the required table permissions.
Reference:
https://docs.databricks.com/aws/en/oltp/projects/data-api
Hope this helps!