https://community.databricks.com/t5/data-engineering/can-i-send-multiple-post-requests-to-an-api-endpoint-and-get-the/m-p/100459#M40308 <P>Hello I am trying to send multiple post requests to an endpoint, i have a spark dataframe and each column of this dataframe is sent through the payload of the post request.</P><P>However when i run this in my notebook, no exception is raised. I'm guessing it is because the requests are executed on spark workers</P><P>That's why i wonder how can i get the info if a post request returned a HTTP error 422 or 502 ?</P><P>Here's something i tried but with no result</P><LI-CODE lang="python">url = "string" headers = { 'header': 'XXXXX' } def process_partition(partition): partition_errors = [] for row in partition: payload = { "col1": str(row["col1"]), } try: response = requests.post(url, json=payload, headers=headers) response.raise_for_status() except requests.exceptions.HTTPError as err: partition_errors.append({"error": str(err), "data": row.asDict()}) except Exception as e: partition_errors.append({"error": str(e), "data": row.asDict()}) return partition_errors def log_errors(errors): if errors: for error in errors: print(f"Error: {error['error']}, Data: {error['data']}") else: print("All data has been sent.") all_errors = [] def collect_partition_errors(partition): errors = process_partition(partition) all_errors.extend(errors) df_websocket_data.foreachPartition(collect_partition_errors) log_errors(all_errors)</LI-CODE><P>&nbsp;</P> Fri, 29 Nov 2024 18:48:03 GMT Layer 2024-11-29T18:48:03Z https://community.databricks.com/t5/data-engineering/can-i-send-multiple-post-requests-to-an-api-endpoint-and-get-the/m-p/100459#M40308 <P>Hello I am trying to send multiple post requests to an endpoint, i have a spark dataframe and each column of this dataframe is sent through the payload of the post request.</P><P>However when i run this in my notebook, no exception is raised. I'm guessing it is because the requests are executed on spark workers</P><P>That's why i wonder how can i get the info if a post request returned a HTTP error 422 or 502 ?</P><P>Here's something i tried but with no result</P><LI-CODE lang="python">url = "string" headers = { 'header': 'XXXXX' } def process_partition(partition): partition_errors = [] for row in partition: payload = { "col1": str(row["col1"]), } try: response = requests.post(url, json=payload, headers=headers) response.raise_for_status() except requests.exceptions.HTTPError as err: partition_errors.append({"error": str(err), "data": row.asDict()}) except Exception as e: partition_errors.append({"error": str(e), "data": row.asDict()}) return partition_errors def log_errors(errors): if errors: for error in errors: print(f"Error: {error['error']}, Data: {error['data']}") else: print("All data has been sent.") all_errors = [] def collect_partition_errors(partition): errors = process_partition(partition) all_errors.extend(errors) df_websocket_data.foreachPartition(collect_partition_errors) log_errors(all_errors)</LI-CODE><P>&nbsp;</P> Fri, 29 Nov 2024 18:48:03 GMT https://community.databricks.com/t5/data-engineering/can-i-send-multiple-post-requests-to-an-api-endpoint-and-get-the/m-p/100459#M40308 Layer 2024-11-29T18:48:03Z https://community.databricks.com/t5/data-engineering/can-i-send-multiple-post-requests-to-an-api-endpoint-and-get-the/m-p/100851#M40441 <P>The return type for foreachPartition is None, so this is expected. If you're looking to do arbitrary code execution and return a result, <A href="https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.mapInPandas.html" target="_self">mapInPandas</A> or&nbsp;<A href="https://spark.apache.org/docs/3.4.2/api/python/reference/pyspark.sql/api/pyspark.sql.functions.pandas_udf.html" target="_self">Pandas UDFs</A> are good choices - you'd want to combine those with something like a .<A href="https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.toLocalIterator.html" target="_self">toLocalIterator</A> call to interact with results in Python</P> Wed, 04 Dec 2024 06:46:59 GMT https://community.databricks.com/t5/data-engineering/can-i-send-multiple-post-requests-to-an-api-endpoint-and-get-the/m-p/100851#M40441 cgrant 2024-12-04T06:46:59Z