cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Concrete way of loading data from APIs into the Databricks sphere (Data Mesh)

samuraidjakk
New Contributor II

We are in the process of implementing data mesh in our organization. When trying to help the different teams produce raw data, the absolute majority want to do this through their APIs. We tried an implementation where we did web-requests directly from Databricks using UDFs, but this did not work very well unfortunately. 

Dealing with APIs must be pretty common, but I find it hard to find resources on this subject. We are considering recommending to set up a cloud function and dump the API response into a storage bucket of some sort. However, I feel like there should exist a "better" or easier way.

We are looking for ways to minimize the workload a team needs to get their data into our platform. So I wonder, what is the best way to deal with API data ingestion into Databricks?

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group