Write streaming output to DynamoDB

sudhanshu1 — Tue, 20 Dec 2022 16:52:09 GMT

Hi All,

I am trying to write a streaming DF into dynamoDB with below code.

tumbling_df.writeStream \

.format("org.apache.spark.sql.execution.streaming.sinks.DynamoDBSinkProvider") \

.option("region", "eu-west-2") \

.option("tableName", "PythonForeachTest") \

.outputMode("update") \

.start()

However i get an error

Py4JJavaError: An error occurred while calling o3880.start.

: java.lang.ClassNotFoundException:

Failed to find data source: org.apache.spark.sql.execution.streaming.sinks.DynamoDBSinkProvider. Please find packages at

Could you please help me to find this package and how to import it please?

Re: Write streaming output to DynamoDB

LandanG — Tue, 20 Dec 2022 20:15:02 GMT

Hi @SUDHANSHU RAJ ,

I can't seem to find much on the "DynamoDBSinkProvider" source. Have you checked out the link for the streaming to DynamoDB documentation?