mapInPandas not working in serverless compute

chinmay0924
New Contributor III

‎08-11-2025 10:53 PM

Using `mapInPandas` in serverless compute (Environment version 2) gives the following error,
```
Py4JError: An error occurred while calling o543.mapInPandas. Trace: py4j.Py4JException: Method mapInPandas([class org.apache.spark.sql.catalyst.expressions.PythonUDF, class java.lang.Boolean]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:352) at py4j.Gateway.invoke(Gateway.java:297) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:197) at py4j.ClientServerConnection.run(ClientServerConnection.java:117) at java.lang.Thread.run(Thread.java:750)
```

0 Kudos

Khaja_Zaffer
Esteemed Contributor

‎08-11-2025 11:26 PM

Hello @chinmay0924

Good day

According to the documentation - (https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/limitations) - This is a limitation of databricks connect. Unfortunately, you have to work with spark.sql or dataframes orSwitch to a standard (non-serverless) all-purpose cluster or job cluster, 

 

I am open to other contributions on this issue. 

 

chinmay0924
New Contributor III
In response to Khaja_Zaffer

‎08-11-2025 11:33 PM

Hello @Khaja_Zaffer 
The documentation you linked does not mention anywhere that mapInPandas is not supported. It says `Only Spark connect APIs are supported. Spark RDD APIs are not supported`. I have not used Spark RDD APIs. All I am trying to do is `dataframe.mapInPandas()` on a spark dataframe.

0 Kudos

Khaja_Zaffer
Esteemed Contributor
In response to chinmay0924

‎08-13-2025 04:30 AM

Hello @chinmay0924 

Are you using 
Serverless compute via notebook UI OR

Serverless compute via Databricks Connect 

0 Kudos

mark_ott
Databricks Employee
Databricks Employee

‎11-11-2025 02:46 AM

The error you are seeing when using mapInPandas in serverless compute with Environment version 2 is due to an incompatibility in the environment's supported Spark features. Specifically, Environment version 2 on serverless compute does not support mapInPandas, which triggers the Py4JException indicating that the method does not exist on the JVM side of Spark in your environment.

Why This Happens

  • Serverless Environment Restrictions
    The version of Spark or configuration for serverless pools (especially with certain environments like Databricks Runtime 11.x or higher) may not expose certain Apache Spark features, including mapInPandas, for security and resource isolation reasons.

  • Method Not Available
    The error message means the Spark JVM backend does not recognize or export the mapInPandas method for remote invocation. It's not your code—it's the compute environment not supporting direct Python UDFs with this Spark construct.

What You Can Do

  • Switch to Standard Compute
    If possible, use a non-serverless, standard compute cluster or an environment version (like Databricks Runtime 9.x or below) where mapInPandas is supported.

  • Use Supported APIs
    In environments where mapInPandas is not supported, use alternatives such as:

    • applyInPandas (on newer versions/environments that support only some Pandas UDFs)

    • Explicit Spark SQL or DataFrame operations

  • Environment Upgrade/Change
    Check if a newer version of the serverless environment, or a configuration update, supports this method since feature support evolves frequently in managed Spark environments.