Databricks Community

longcao · 07-01-2016

Hi there,I'm just getting started with Spark and I've got a moderately sized DataFrame created from collating CSVs in S3 (88 columns, 860k rows) that seems to be taking an unreasonable amount of time to insert (using SaveMode.Append) into Postgres. I...

longcao · 07-28-2016

Hmm, not entirely sure why the snippet might not work for your setup. Is it possible that you are running on a PostgreSQL version that's not compatible with the COPY command in the snippet? I was running PostgreSQL 9.5.x on RDS. As for the flatten, ...

longcao · 07-11-2016

In case anyone was curious how I worked around this, I ended up dropping down to Postgres JDBC and using CopyManager to COPY rows in directly from Spark: https://gist.github.com/longcao/bb61f1798ccbbfa4a0d7b76e49982f84

longcao · 07-01-2016

I'm not sure how to get it to nicely format the code into monospaced, syntax highlighted blocks. The </> code button doesn't seem to work. If anyone knows let me know!

Databricks Community

User Stats

User Activity

Writing DataFrame to PostgreSQL via JDBC extremely slow (Spark 1.6.1)

Re: Writing DataFrame to PostgreSQL via JDBC extremely slow (Spark 1.6.1)

Re: Writing DataFrame to PostgreSQL via JDBC extremely slow (Spark 1.6.1)

Re: Writing DataFrame to PostgreSQL via JDBC extremely slow (Spark 1.6.1)