Databricks Community

chinmay0924 · 08-11-2025

Using `mapInPandas` in serverless compute (Environment version 2) gives the following error,```Py4JError: An error occurred while calling o543.mapInPandas. Trace: py4j.Py4JException: Method mapInPandas([class org.apache.spark.sql.catalyst.expressions...

chinmay0924 · 06-15-2025

```File "/databricks/spark/python/pyspark/sql/pandas/serializers.py", line 346, in _create_array return pa.Array.from_pandas( ^^^^^^^^^^^^^^^^^^^^^ File "pyarrow/array.pxi", line 1126, in pyarrow.lib.Array.from_pandas File "pyarrow/array.pxi", line 3...

chinmay0924 · 06-05-2025

https://docs.databricks.com/api/workspace/functions/createThis documentation gives the sample request payload, and one of the fields is type_json, and there is very little explanation of what is expected in this field. What am I supposed to pass here...

chinmay0924 · 04-30-2025

I am trying to execute pandas UDF in databricks. It gives me the following error on serverless compute,File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b11ff17c-9b25-4ccb-927d-06a7d1ca7221/lib/python3.11/site-packages/pyspark/sql/connect/client/core.p...

chinmay0924 · 09-25-2024

I want to be able to access the RDD methods of a Dataframe, but it seems that this is not supported in spark connect. I have been trying to disable spark connect in the spark config using,spark.databricks.service.server.enabled false but when I check...

chinmay0924 · 08-11-2025

Hello @Khaja_Zaffer The documentation you linked does not mention anywhere that mapInPandas is not supported. It says `Only Spark connect APIs are supported. Spark RDD APIs are not supported`. I have not used Spark RDD APIs. All I am trying to do is ...

chinmay0924 · 06-15-2025

The data in one of the columns is modified in the function that is passed to `mapInPandas`,def df_process_function(df): schema = df.schema def partition_process(iterator): for pdf in iterator: processed_records = partition_fun...

chinmay0924 · 05-06-2025

Thanks a lot for the reply! Final question: Is it possible for the end user to select a serverless client image version or is that not something that I can choose? If so, how? I couldn't find any documentation regarding this.

chinmay0924 · 05-01-2025

If I am using the databricks notebook with serverless attached, how do I change the client side python? I have control over client side python if I use databricks connect from my local setup, but is there a way to change the client side python versio...

chinmay0924 · 09-26-2024

I want to disable and I tried setting spark.databricks.service.server.enabled to false in the spark config while creating the compute, but when I run the notebook with this compute and I do type(spark), I get pyspark.sql.connect.session.SparkSession.

Databricks Community

User Stats

User Activity

mapInPandas not working in serverless compute

mapInPandas returning an intermittent error related to data type interconversion

How to create a function using the functions API in databricks?

Spark connect client and server versions should be same for executing UDFs

How to disable spark connect in the databricks compute?

Re: mapInPandas not working in serverless compute

Re: mapInPandas returning an intermittent error related to data type interconversion

Re: Spark connect client and server versions should be same for executing UDFs

Re: Spark connect client and server versions should be same for executing UDFs

Re: How to disable spark connect in the databricks compute?