Python UDF support in Unity Catalog and runtime 13.3?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2025 11:23 AM
Hi community,
I am running Databricks Unity Catalog. In the DataBricks UI, I see the Policy "shared-gp-(r6g)-small" and Runtime 13.3. (I have access to larger instances, just running a PoC on a small instance).
Can anyone explain what looks like an inconsistency between the documentation and what I am seeing?
The documentation here states that Python UDFs are supported in a cluster running Databricks Runtime 13.3 LTS or above.
On this page, there is sample code to create a UDF in Python.
This code take from that page works for me:
from pyspark.sql.types import LongType
def squared_typed(s):
return s * s
spark.udf.register("squaredWithPython", squared_typed, LongType())
spark.range(1, 20).createOrReplaceTempView("test")
I get an error:
AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): org.apache.spark.sql.catalyst.expressions.PythonUDF are not supported in Unity Catalog. ;
However, when I run the next snippet of sample code from that page
from pyspark.sql.functions import udf
from pyspark.sql.types import LongType
squared_udf = udf(squared, LongType())
df = spark.table("test")
display(df.select("id", squared_udf("id").alias("id_squared")))
from pyspark.sql.functions import udf
from pyspark.sql.types import LongType
squared_udf = udf(squared, LongType())
df = spark.table("test")
display(df.select("id", squared_udf("id").alias("id_squared")))
I get an error:
AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): org.apache.spark.sql.catalyst.expressions.PythonUDF are not supported in Unity Catalog. ;
Is there something I'm missing about how I need to run this sampe code?
