topic Re: Reduce downtime of Postgres table - JDBC overwrite job in Data Engineering

Reduce downtime of Postgres table - JDBC overwrite job

siddharthk — Thu, 08 Jun 2023 19:45:31 GMT

I want to overwrite a Postgresql table transactionStats which is used by the customer facing dashboards.

This table needs to be updated every 30 mins. I am writing a AWS Glue Spark job via JDBC connection to perform this operation.

Spark dataframe write snippet -

df.write.format("jdbc").option("url", "connection_params").option("dbtable","public.transactionStats"). mode("overwrite").save()

I am seeing a downtime on the dashboard while the table is being overwritten

how to avoid downtime? Any solutions? Staging table?

Re: Reduce downtime of Postgres table - JDBC overwrite job

-werners- — Fri, 09 Jun 2023 10:53:57 GMT

can you use rename table? so load a temp table, rename target table to something else, rename temp table to target table.

Re: Reduce downtime of Postgres table - JDBC overwrite job

Anonymous — Thu, 15 Jun 2023 07:09:32 GMT

Hi @Siddharth Kanojiya

We haven't heard from you since the last response from @werners (Customer) . Kindly share the information with us, and in return, we will provide you with the necessary solution.

Thanks and Regards