topic Am trying to use SQL, but createOrReplaceTempView("myDataView") fails in Data Engineering

Am trying to use SQL, but createOrReplaceTempView("myDataView") fails

MikeBrewer — Mon, 13 Jul 2020 11:59:04 GMT

Am trying to use SQL, but createOrReplaceTempView("myDataView") fails.

I can create and display a DataFrame fine...

import pandas as pd
df = pd.DataFrame(['$3,000,000.00','$3,000.00', '$200.5', '$5.5'], columns = ['Amount'])
df

I add another cell, but it fails...

df.createOrReplaceTempView("myDataView")

I get this error..

'DataFrame' object has no attribute 'createOrReplaceTempView'

I see this example out there on the net allot, but don't understand why it fails for me. I am using

Community edition. 6.5 (includes Apache Spark 2.4.5, Scala 2.11)

-werners- — Wed, 15 Jul 2020 12:09:53 GMT

I never worked with pandas on spark, but a pandas dataframe is not the same as a spark dataframe.

You need to convert it to a spark dataframe first with Koalas f.e.

acorson — Wed, 15 Jul 2020 19:15:36 GMT

You need to convert the pandas DF to an spark DF. Enabling Apache Arrow will make this process faster.

import numpy as np import pandas as pd

Enable Arrow-based columnar data transfers

park.conf.set("spark.sql.execution.arrow.enabled", "true")

Generate a pandas DataFrame

df = pd.DataFrame(np.random.rand(100, 3))

Create a Spark DataFrame from a pandas DataFrame using Arrow

f = spark.createDataFrame(pdf)

Convert the Spark DataFrame back to a pandas DataFrame using Arrow

esult_pdf = df.select("*").toPandas()

sachinthana — Thu, 01 Jul 2021 17:10:37 GMT

This is worked for me. Thank you @acorson