mark_ott
Databricks Employee
Databricks Employee

‎11-10-2025 05:27 PM

You're encountering a common limitation when trying to use an external HTTP request (like the Python requests library) inside a Unity Catalog UDF in Databricks. While your code is correct for a regular notebook environment, Unity Catalog UDFs (and, similarly, Spark SQL UDFs running in distributed compute) run in a restricted, sandboxed environment where outbound HTTP calls are not allowed for security reasons.

Why It Works in Notebook But Not in UDF

  • Notebooks: Full access to Python environment, packages, and outbound internet.

  • Unity Catalog Functions: Sandboxed, limited. No network access, no additional installs, can only use a subset of Python's standard library.

This is by design, because otherwise user-defined SQL functions could be used to exfiltrate data or trigger security vulnerabilities.

What Are Your Options?

Use Notebook Code for HTTP Calls

  • Perform the HTTP call from your notebook code (or Python UDF).

  • If the workflow requires SQL, submit results into a table and consume from there.

Use External Functions (Databricks Feature)

Databricks External Functions allow you to call external HTTP endpoints securely from SQL. These are designed for invoking webhooks or external APIs from SQL, similar to what you're after, and are most often integrated with Unity Catalog.

Basic example:

sql
CREATE EXTERNAL FUNCTION send_teams_message(url STRING, payload STRING) RETURNS STRING USING REQUEST_URL '<your-azure-function-or-api-endpoint>';

You need a secure HTTP endpoint (Azure Function, AWS Lambda, REST API you manage) to receive requests from Databricks.

Why "No Errors"?

  • Unity Catalog UDFs may fail silently for forbidden or unimplemented features.

  • If outbound HTTP is restricted, requests.post() simply does not run, and you may get None or an empty response, with no error bubbling to the surface.

Summary Table

Environment Can Do HTTP Requests? Example Approach
Notebook Python Yes Use requests or similar
Python UDF No (in UC/SparkSQL) Not supported
Unity Catalog UDF No Not supported
External Function Yes Use Databricks External Function
 
 

What To Do Next

  • Refactor: Move the message send logic out of the UC UDF.

  • Use Databricks External Functions if you need SQL access to HTTP APIs.

  • If working in notebooks, use Python code directly where possible.

0 Kudos