Databricks Community

Dataninsight · Sunday

A Complete Guide to Scala and Java UDFs in Unity Catalog

Imagine your company stores customer phone numbers from around the world. Some come in as (415) 555-0132, others as +44 20 7946 0958, others as 020-7946-0958. They all mean something valid, but no two look the same which makes it nearly impossible to deduplicate customers, send SMS reminders through a third-party provider, or join phone-based records across systems.

The fix already exists: a well-tested Scala or Java library called libphonenumber (built by Google) knows how to parse almost any phone number format on Earth and convert it into one consistent standard, called E.164 (e.g. +14155550132). The problem isn't the logic, it's that this logic lives in a JVM library, and your analysts query everything in SQL.

This guide walks through solving that problem end to end: writing the logic once, registering it in Databricks Unity Catalog, and making it callable by anyone on your team who knows SQL with no Scala knowledge required on their end.

A few terms to know:

UDF (User-Defined Function) A custom function you write yourself, because the built-in SUM, UPPER, ROUND functions don't do what you need.
Unity Catalog Databricks' governance layer: it controls who can see and run what, across your whole organization.
JAR A single packaged file containing compiled Java/Scala code (and, if it's a "JAR," all the external libraries it depends on too).
Volume A managed folder inside Unity Catalog where you can store files like JARs, so Databricks compute can access them.
Handler The exact method Databricks should call when your UDF runs you point at it by its full package path.

If you’ve never touched Scala before, that’s fine you’ll copy the handler code as-is. The part you’ll actually interact with day to day is the SQL at the very end.

Why this needs to be a governed UDF, not just a notebook function

Before Unity Catalog UDFs existed, you had two bad options:

Define the function in every notebook. It disappears when the session ends. Nobody outside that notebook can use it. If the logic needs a fix, you’re hunting down every copy of it.
Rewrite it in Python. Possible, but risky libphonenumber's Python port lags behind the Java version, and rewriting battle-tested logic introduces new bugs.

Registering it in Unity Catalog instead means:

You write it once. Everyone, SQL users, notebook users, job authors calls the same function.
It’s governed. You decide exactly who can run it, via standard permission grants.
It’s discoverable. Anyone can find it in Catalog Explorer instead of asking around on Slack.
It works everywhere. classic compute, serverless notebooks and jobs, and SQL warehouses.

What you’ll need before starting

A Databricks workspace with Unity Catalog enabled
USAGE and CREATE FUNCTION permission on the target schema, and USAGE on the catalog
A local machine with JDK 17 and sbt installed (for building the JAR)

Scala 2.13.16 (Scala 2.12 isn’t supported)
A Unity Catalog volume to store the JAR, and READ VOLUME permission on it (you'll create this in Step 3 if it doesn't exist)

Step 1: Write the function

Every Unity Catalog Scala UDF has one rule that trips people up the first time:

the handler must be a method on a Scala object, not a class. Create a new file at src/main/scala/com/acme/udf/PhoneUDF.scala:

%scala
package com.acme.udf
import com.google.i18n.phonenumbers.PhoneNumberUtil
object PhoneUDF {
  private val util = PhoneNumberUtil.getInstance()
  def toE164(raw: String, region: String): String = {
    if (raw == null) return null
    val parsed = util.parse(raw, region)
    util.format(parsed, PhoneNumberUtil.PhoneNumberFormat.E164)
  }
}

What this does, in plain terms: toE164 takes a messy phone number plus the country it's from (so it knows how to interpret the local format), and hands back the clean, standardized version.

The if (raw == null) return null line matters, without it, a single blank phone number in your dataset could crash the whole query.

Step 2: Tell your build tool about the dependency

The PhoneNumberUtil class comes from Google's libphonenumber library, not from Scala's standard library so your project needs to know to include it. Create or edit build.sbt in your project root:

%scala
scalaVersion := "2.13.16"
ThisBuild / organization := "com.acme"
lazy val phoneUdf = (project in file("."))
  .settings(
    name := "phone-udf",
    libraryDependencies += "com.googlecode.libphonenumber" % "libphonenumber" % "8.13.35"
  )

Step 3: Enable the plugin that bundles everything together

Databricks needs a single, self-contained file: your code and
every library it depends on called a “JAR.”

Create project/assembly.sbt:

%scala
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.0.0")

Step 4: Build the JAR

From your project’s root directory, run:

sbt clean assembly

This produces a file like target/scala-2.13/phone-udf-assembly-0.1.0.jar. That single file now contains your handler code and the libphonenumber library it needs nothing else to install later.

Step 5: Upload the JAR to a Unity Catalog volume

If you don’t already have a volume to store JARs, create one:

CREATE VOLUME IF NOT EXISTS acme.telco.jars
COMMENT 'Storage for UDF JAR files';

Then upload the file itself through the UI:

In your Databricks workspace, open Catalog Explorer.
Navigate to the catalog and schema containing your volume (acme → telco → jars).
Click Upload to this volume and select your JAR file.
Once uploaded, click the file name, then Copy path.

You’ll get a path that looks like: /Volumes/acme/telco/jars/phone-udf-assembly-0.1.0.jar

Keep that path handy, you need it in the next step.

Step 6: Register the function in Unity Catalog

This is the step that turns your JAR into something callable from SQL:

CREATE OR REPLACE FUNCTION acme.telco.to_e164(raw STRING, region STRING)
RETURNS STRING
LANGUAGE SCALA
ENVIRONMENT (
  java_dependencies = '["/Volumes/acme/telco/jars/phone-udf-assembly-0.1.0.jar"]',
  environment_version = '4'
)
HANDLER 'com.acme.udf.PhoneUDF.toE164';

Note: If getting syntax error for now wait for some days.

Breaking down what each part actually means:

acme.telco.to_e164 : the full name your function will have in SQL: catalog, schema, function name.
RETURNS STRING : must match the handler's return type exactly.
ENVIRONMENT : this is where you point at the JAR you just uploaded, and declare environment_version = '4', which tells Databricks to use Scala 2.13.16 and JDK 17 to run it (required for Scala/Java UDFs).
HANDLER : the exact object and method path: package.Object.method.

You need USAGE + CREATE FUNCTION on the telco schema, and USAGE on the acme catalog, to run this successfully.

Step 7: Decide who’s allowed to use it

By default, only you (the creator) can run the new function. Grant access to whoever needs it:

GRANT EXECUTE ON FUNCTION acme.telco.to_e164 TO `data-analysts`;

They’ll also need USAGE on the acme catalog and telco schema, without that, EXECUTE alone won't let them call it.

Step 8: Actually use it

This is the payoff, from here on, nobody needs to think about Scala, JARs, or volumes again. It’s just SQL:

-- Call it directly
SELECT id, to_e164(phone, country_code) AS phone_e164
FROM acme.telco.customers;

For a cleaner handoff to other teams, wrap it in a view so they never even see the raw function call:

CREATE OR REPLACE VIEW acme.telco.customers_clean AS
SELECT id, to_e164(phone, country_code) AS phone_e164
FROM acme.telco.customers;

Now anyone can simply query acme.telco.customers_clean and get clean, standardized phone numbers with the underlying logic fully governed and auditable.

Step 9: Updating the logic later

Business logic changes. When it does, you don’t touch the function definition you rebuild and replace:

Update your Scala code.
Rebuild with a new version number: sbt clean assembly → phone-udf-assembly-0.2.0.jar.
Upload the new JAR to the same volume.
Re-run CREATE OR REPLACE FUNCTION, pointing java_dependencies at the new path.

No cluster restart needed: the next call to the function picks up the new code automatically.

Common issues to watch for

“Function not found” after registering double check you’re using the fully qualified name (catalog.schema.function), and that you have USAGE on both the catalog and schema.
Registration fails on classic compute confirm you’re on Databricks Runtime 18.2 or above; earlier versions don’t support this feature.
UDF returns NULL unexpectedly if a parameter is a primitive type (like Int), a NULL input silently short-circuits to NULL output. Wrap it in Option[Int] if you need to handle nulls explicitly yourself.
JAR built but “class not found” at call time this usually means the JAR wasn’t built as a JAR (i.e., the assembly plugin wasn’t set up correctly), so the dependency isn’t actually bundled inside it.

Why this is worth doing

None of this is complicated once you’ve done it once: write the function, package it, upload it, register it, grant access, call it. What it makes you is bigger than the mechanics:

The same phone-normalization logic runs everywhere no more three slightly-different Python reimplementations across three teams.
It’s auditable: one place to see who can run it, one JAR to version.
The people using it in SQL every day never need to learn Scala, understand JARs, or know a volume exists. They just call to_e164(...) like it was always part of SQL.