Hello,
IMHO, having a HTTP related task in a Spark cluster is an anti-pattern. This kind of code executes at the Driver, it will be synchronous and adds overhead. This is one of the reasons, DLT (or SDP - Spark Declarative Pipeline) does not have REST based tasks.

Please review if this task can be done outside Databricks like below,
1) Event based trigger: push the result from Databricks to cloud storage; and this creates an event (Event grid) to a listener like Function/Logic App that will perform HTTP task
2) Classic Poller: Azure Function App to check for an expectation every 'n' mins. if met; execute the HTTP task