topic Re: Unity Catalog Pandas on Spark Limitation in Administration & Architecture

Unity Catalog Pandas on Spark Limitation

RamlaSuhra — Fri, 28 Jun 2024 17:49:30 GMT

According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.

1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Pandas on Spark are not supported.

Why is Pandas on Spark not supported in "Shared" Access mode clusters when they are UC enabled?

Is there a feature getting added to support this soon?

Re: Unity Catalog Pandas on Spark Limitation

raphaelblg — Tue, 02 Jul 2024 14:26:23 GMT

@RamlaSuhra, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished.

UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on Graviton clusters. UDTFs are still under development.

A shared cluster architecture differs extensively from the single-user cluster, and I would say it's way more complicated, that's why it takes more time to receive some features.