I cannot see the output when using pandas_api() on spark dataframe

kaushalshelat
Databricks Partner

‎04-29-2025 09:44 AM

Hi all, I started learning spark and databricks recently along with python. while running below line of code it did not throw any error and seem to run ok but didn't show me output either.

test=cust_an_inc1.pandas_api()
test.show()
where cust_an_inc1 is spark dataframe.
The message i get is as below
 
 
'/databricks/python/lib/python3.11/site-packages/pyspark/pandas/utils.py:1035: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is set to True. This can cause unexpected behavior from pandas API on Spark since pandas API on Spark follows the behavior of pandas, not SQL. warnings.warn(message, PandasAPIOnSparkAdviceWarning)'
 
so no error but no output either.
Can anyone know issue with this please. Thanks in advance.

 

samshifflett46
Databricks Partner

‎04-29-2025 06:00 PM

Hey @kaushalshelat, I don't think you need to call '.show()' in order to print the DF out. You should just be able to print by writing the df variable name.

Docs

pandas_api.png

RiyazAliM
Honored Contributor

‎04-30-2025 03:11 AM

Hi @kaushalshelat 

Ideally, `test.show()` should've thrown an error as test is the pandas dataframe now.

`.show()` is a spark df method and wouldn't work with pandas.

If you want to see a subset of the data try `.head()` or `.tail(n)` rather then `.show()`.

Thanks,

Riz

View solution in original post