Ingest data from REST endpoint into Databricks

RodrigoE — Thu, 23 Apr 2026 20:10:44 GMT

Hello,

I'm looking for the best option to retrieve between 1-1.5TB of data per day from a REST API into Databricks.

Thank you,

Rodrigo Escamilla

Re: Ingest data from REST endpoint into Databricks

Ashwin_DSA — Thu, 23 Apr 2026 21:10:08 GMT

Hi @RodrigoE,

It would be helpful to have additional information to recommend the best options for your scenario.

Who owns the REST API?
Is that in your control?
Can the source push data to Databricks, or should you pull on a schedule?

If the source can push the data, consider Zerobus. This is the cleanest, most scalable Databricks-native pattern if the producer is under your control.

If you have no control over the source, you can build a custom Python data source wrapping their REST API and run it as a Databricks job/stream. While the pattern will work for your volumes, the bottleneck is usually the API’s own throughput/limits, not Databricks.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

topic Ingest data from REST endpoint into Databricks in Data Engineering

Ingest data from REST endpoint into Databricks

Re: Ingest data from REST endpoint into Databricks