Migrating data from delta lake to RDS MySQL and ElasticSearch

[AmanSehgal]

There are mechanisms (like DMS) to get data from RDS to delta lake and store the data in parquet format, but is it possible to reverse of this in AWS?

I want to send data from data lake to MySQL RDS tables in batch mode.

And the next step is to send the data to ElasticSearch.

[Hubert-Dudek]

@Aman Sehgal​ , In Azure, for such migrations, I used Azure Data Factory. It was just table copy and data copy (so no indexes). In AWS equivalent is AWS glue but I don't have experience in practice with that. In Azure Data Factory, there is a connector to S3 and Mysql so that you could try it with a multi-cloud approach.

Eventually, you can write your script on Databricks. If it is not a one-time copy but regular, I probably would sit and write all logic - list tables, create tables, register tables in hive metastore, copy data between them, etc.

[Kaniz_Fatma]

Hi @Aman Sehgal​ , Just a friendly follow-up. Do you still need help, or @Hubert Dudek​ 's response help you to find the solution? Please let us know.

[AmanSehgal]

@Kaniz Fatma​  and @Hubert Dudek​  - writing to MySQL RDS is relatively simpler. I'm finding ways to export data into Elasticsearch

[Kaniz_Fatma]

Thanks for the quick update @Aman Sehgal​ . Let's find an answer for you.