cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Unity Catalog Pandas on Spark Limitation

RamlaS
New Contributor II

According to Databricks UC Documentation, below are the some of the limitations on Shared Mode Cluster.

1. In Databricks Runtime 13.3 LTS and above, Python scalar UDFs and Pandas UDFs are supported. Other Python UDFs, including UDAFs, UDTFs, and Pandas on Spark are not supported.

Why is Pandas on Spark not supported in "Shared" Access mode clusters when they are UC enabled?

Is there a feature getting added to support this soon?

1 ACCEPTED SOLUTION

Accepted Solutions

raphaelblg
Databricks Employee
Databricks Employee

@RamlaS, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished. 

UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on Graviton clusters. UDTFs are still under development. 

A shared cluster architecture differs extensively from the single-user cluster, and I would say it's way more complicated, that's why it takes more time to receive some features.

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

View solution in original post

1 REPLY 1

raphaelblg
Databricks Employee
Databricks Employee

@RamlaS, UDFs on Unity Catalog is a feature that, at the current moment is still on the Public Preview stage. This means that the development has yet not finished. 

UDFs can be used on DBR 13.3 and above, UDAFs are already available for DBR 15.2 on Graviton clusters. UDTFs are still under development. 

A shared cluster architecture differs extensively from the single-user cluster, and I would say it's way more complicated, that's why it takes more time to receive some features.

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group