cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unity Catalog Pandas on Spark Limitation

RamlaSuhra
New Contributor II

According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.

1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Pandas on Spark are not supported.

Why is Pandas on Spark not supported in "Shared" Access mode clusters when they are UC enabled?

Is there a feature getting added to support this soon?

1 ACCEPTED SOLUTION

Accepted Solutions

raphaelblg
Databricks Employee
Databricks Employee

@RamlaSuhra, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished. 

UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on Graviton clusters. UDTFs are still under development. 

A shared cluster architecture differs extensively from the single-user cluster, and I would say it's way more complicated, that's why it takes more time to receive some features.

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

1 REPLY 1

raphaelblg
Databricks Employee
Databricks Employee

@RamlaSuhra, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished. 

UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on Graviton clusters. UDTFs are still under development. 

A shared cluster architecture differs extensively from the single-user cluster, and I would say it's way more complicated, that's why it takes more time to receive some features.

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now