topic Re: Can we use spark-stream to read/write data from mysql? I can't find an example. in Data Engineering

Can we use spark-stream to read/write data from mysql? I can't find an example.

sarvesh — Fri, 19 Nov 2021 08:08:47 GMT

If someone can link me an example where stream is used to read or write to mysql please do.

Re: Can we use spark-stream to read/write data from mysql? I can't find an example.

sarvesh — Fri, 19 Nov 2021 11:04:48 GMT

Thank you for taking sometime out to reply, as of now i have not found a structured way.

Re: Can we use spark-stream to read/write data from mysql? I can't find an example.

Hubert-Dudek — Sat, 20 Nov 2021 16:01:52 GMT

Regarding writing (sink) is possible without problem via foreachBatch .

I use it in production - stream autoload csvs from data lake and writing foreachBatch to SQL (inside foreachBatch function you have temporary dataframe with records and just use write to any jdbc or odbc).

Here is more deltails:

https://docs.databricks.com/spark/latest/structured-streaming/foreach.html

Reading stream from mysql is not the best architecture and officialy is not possible. Theoretically you could create custom reciver but better idea is just put what you save to mysql to kafka or some other broker/queue.

Easy way around (but it make sense only if you have less new records than 1 per second) is to use Azure logic apps - new record in Mysql is trigger and append data to Event hub which is then read as a stream by Databricks.

Re: Can we use spark-stream to read/write data from mysql? I can't find an example.

jose_gonzalez — Mon, 22 Nov 2021 20:02:08 GMT

Great example of your use case @Hubert Dudek