mapInPandas not working in serverless compute

[chinmay0924]

Using `mapInPandas` in serverless compute (Environment version 2) gives the following error,
```
Py4JError: An error occurred while calling o543.mapInPandas. Trace: py4j.Py4JException: Method mapInPandas([class org.apache.spark.sql.catalyst.expressions.PythonUDF, class java.lang.Boolean]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:352) at py4j.Gateway.invoke(Gateway.java:297) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:197) at py4j.ClientServerConnection.run(ClientServerConnection.java:117) at java.lang.Thread.run(Thread.java:750)
```

[Khaja_Zaffer]

Hello @chinmay0924

Good day

According to the documentation - (https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/limitations) - This is a limitation of databricks connect. Unfortunately, you have to work with spark.sql or dataframes orSwitch to a standard (non-serverless) all-purpose cluster or job cluster, 

I am open to other contributions on this issue. 

[chinmay0924]

Hello @Khaja_Zaffer 
The documentation you linked does not mention anywhere that mapInPandas is not supported. It says `Only Spark connect APIs are supported. Spark RDD APIs are not supported`. I have not used Spark RDD APIs. All I am trying to do is `dataframe.mapInPandas()` on a spark dataframe.

[Khaja_Zaffer]

Hello @chinmay0924 

Are you using 
Serverless compute via notebook UI OR

Serverless compute via Databricks Connect