topic Re: I cannot see the output when using pandas_api() on spark dataframe in Data Engineering

I cannot see the output when using pandas_api() on spark dataframe

kaushalshelat — Tue, 29 Apr 2025 16:44:56 GMT

Hi all, I started learning spark and databricks recently along with python. while running below line of code it did not throw any error and seem to run ok but didn't show me output either.

test=cust_an_inc1.pandas_api()

test.show()

where cust_an_inc1 is spark dataframe.

The message i get is as below

'/databricks/python/lib/python3.11/site-packages/pyspark/pandas/utils.py:1035: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is set to True. This can cause unexpected behavior from pandas API on Spark since pandas API on Spark follows the behavior of pandas, not SQL. warnings.warn(message, PandasAPIOnSparkAdviceWarning)'

so no error but no output either.

Can anyone know issue with this please. Thanks in advance.

Re: I cannot see the output when using pandas_api() on spark dataframe

samshifflett46 — Wed, 30 Apr 2025 01:00:41 GMT

Hey @kaushalshelat, I don't think you need to call '.show()' in order to print the DF out. You should just be able to print by writing the df variable name.

Docs

Re: I cannot see the output when using pandas_api() on spark dataframe

RiyazAliM — Wed, 30 Apr 2025 10:11:38 GMT

Hi @kaushalshelat

Ideally, `test.show()` should've thrown an error as test is the pandas dataframe now.

`.show()` is a spark df method and wouldn't work with pandas.

If you want to see a subset of the data try `.head()` or `.tail(n)` rather then `.show()`.

Thanks,