https://community.databricks.com/t5/data-engineering/how-to-output-data-from-databricks/m-p/63900#M32392 <P>Hello,</P><P>I am just starting with Databricks in Azure and I need to output the data to an Affinity CRM system.</P><P>Affinity has an API and I am wondering is there any sort of automated / data pipeline sort of way to tell databricks to just pump the data into Affinity or do I have to write some sort of Python script and make it run in Databricks?</P><P>Alternatively I could create a microservice that gets the data from Databricks and sends it to Affinity but that seems like the worst options to me.</P><P>I just don't know how the entire thing works.</P> Sat, 16 Mar 2024 13:23:42 GMT Abdul1 2024-03-16T13:23:42Z https://community.databricks.com/t5/data-engineering/how-to-output-data-from-databricks/m-p/63900#M32392 <P>Hello,</P><P>I am just starting with Databricks in Azure and I need to output the data to an Affinity CRM system.</P><P>Affinity has an API and I am wondering is there any sort of automated / data pipeline sort of way to tell databricks to just pump the data into Affinity or do I have to write some sort of Python script and make it run in Databricks?</P><P>Alternatively I could create a microservice that gets the data from Databricks and sends it to Affinity but that seems like the worst options to me.</P><P>I just don't know how the entire thing works.</P> Sat, 16 Mar 2024 13:23:42 GMT https://community.databricks.com/t5/data-engineering/how-to-output-data-from-databricks/m-p/63900#M32392 Abdul1 2024-03-16T13:23:42Z https://community.databricks.com/t5/data-engineering/how-to-output-data-from-databricks/m-p/63912#M32401 <P>We need more info on what kind of data, volume and what the called APi can handle. Calling an API for single records in parallel can be achieved using UDF(see <A href="https://community.databricks.com/t5/data-engineering/performance-issue-with-pyspark-udf-function-calling-rest-api/td-p/6890" target="_self">THIS</A>). You need to be careful to batch the records so that the target API can handle the parallel load. If you want to send an entire file via API (assuming file size is within the API limits), you can use Synapse pipeline activity(assuming you are using Azure). Databricks does not have any inbuilt feature for this as far as I know.</P> Sun, 17 Mar 2024 01:06:43 GMT https://community.databricks.com/t5/data-engineering/how-to-output-data-from-databricks/m-p/63912#M32401 Edthehead 2024-03-17T01:06:43Z