I cannot see the output when using pandas_api() on spark dataframe

[kaushalshelat]

Hi all, I started learning spark and databricks recently along with python. while running below line of code it did not throw any error and seem to run ok but didn't show me output either.

test=cust_an_inc1.pandas_api()

test.show()

where cust_an_inc1 is spark dataframe.

The message i get is as below

'/databricks/python/lib/python3.11/site-packages/pyspark/pandas/utils.py:1035: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is set to True. This can cause unexpected behavior from pandas API on Spark since pandas API on Spark follows the behavior of pandas, not SQL. warnings.warn(message, PandasAPIOnSparkAdviceWarning)'

so no error but no output either.

Can anyone know issue with this please. Thanks in advance.

[samshifflett46]

Hey @kaushalshelat, I don't think you need to call '.show()' in order to print the DF out. You should just be able to print by writing the df variable name.

Docs

[aayrm5 · Accepted solution]

Hi @kaushalshelat 

Ideally, `test.show()` should've thrown an error as test is the pandas dataframe now.

`.show()` is a spark df method and wouldn't work with pandas.

If you want to see a subset of the data try `.head()` or `.tail(n)` rather then `.show()`.

Thanks,