https://community.databricks.com/t5/data-engineering/rest-api-call/m-p/69214#M33860 <P>There is a requirement to create a pipeline that calls a rest Api and we have to store the data in datawarehouse which is the best was to do this operation?</P> Fri, 17 May 2024 05:32:06 GMT Sushmg 2024-05-17T05:32:06Z https://community.databricks.com/t5/data-engineering/rest-api-call/m-p/69214#M33860 <P>There is a requirement to create a pipeline that calls a rest Api and we have to store the data in datawarehouse which is the best was to do this operation?</P> Fri, 17 May 2024 05:32:06 GMT https://community.databricks.com/t5/data-engineering/rest-api-call/m-p/69214#M33860 Sushmg 2024-05-17T05:32:06Z https://community.databricks.com/t5/data-engineering/rest-api-call/m-p/69220#M33863 <P>there are several ways to do this.<BR />you could use python (or scala or ...) to do the call. then transform it and write it to the dwh.<BR />Or you could do the call, write the raw data and process it later on.<BR />Or you could use an ETL/ELT tool that can do the rest call (so you do not have to program it using python or whatever) and transform the data with databricks or some other tool.<BR />Basically there is no best way, it depends on your needs and the functionality of the API itself and the tools you can use.<BR />But it is definitely possible.</P> Fri, 17 May 2024 07:08:48 GMT https://community.databricks.com/t5/data-engineering/rest-api-call/m-p/69220#M33863 -werners- 2024-05-17T07:08:48Z