topic Pandas dataframe to a table in Data Engineering

Pandas dataframe to a table

KiranRastogi — Mon, 08 May 2017 06:55:01 GMT

I want to write a pandas dataframe to a table, how can I do this ? Write command is not working, please help.

Re: Pandas dataframe to a table

PaulLintilhac — Mon, 25 Sep 2017 06:50:29 GMT

could you post the code for the "write" command you are using?

Re: Pandas dataframe to a table

amy_wang — Wed, 27 Sep 2017 18:13:12 GMT

Hey Kiran,

Just taking a stab in the dark but do you want to convert the Pandas DataFrame to a Spark DataFrame and then write out the Spark DataFrame as a non-temporary SQL table?

import pandas as pd
## Create Pandas Frame
pd_df = pd.DataFrame({u'2017-01-01': 1, u'2017-01-02': 2}.items())
## Convert into Spark DataFrame
spark_df = spark.createDataFrame(pd_df)
## Write Frame out as Table
spark_df.write.mode("overwrite").saveAsTable("db.table_name")

The Python documentation for

saveAsTable

is also available here: Python API Documentation.

Amy