topic Re: Migrating data from delta lake to RDS MySQL and ElasticSearch in Data Engineering

Migrating data from delta lake to RDS MySQL and ElasticSearch

AmanSehgal — Thu, 21 Apr 2022 04:44:53 GMT

There are mechanisms (like DMS) to get data from RDS to delta lake and store the data in parquet format, but is it possible to reverse of this in AWS?

I want to send data from data lake to MySQL RDS tables in batch mode.

And the next step is to send the data to ElasticSearch.

Re: Migrating data from delta lake to RDS MySQL and ElasticSearch

Hubert-Dudek — Thu, 21 Apr 2022 09:20:11 GMT

@Aman Sehgal , In Azure, for such migrations, I used Azure Data Factory. It was just table copy and data copy (so no indexes). In AWS equivalent is AWS glue but I don't have experience in practice with that. In Azure Data Factory, there is a connector to S3 and Mysql so that you could try it with a multi-cloud approach.

Eventually, you can write your script on Databricks. If it is not a one-time copy but regular, I probably would sit and write all logic - list tables, create tables, register tables in hive metastore, copy data between them, etc.

Re: Migrating data from delta lake to RDS MySQL and ElasticSearch

AmanSehgal — Tue, 26 Apr 2022 12:05:28 GMT

@Kaniz Fatma and @Hubert Dudek - writing to MySQL RDS is relatively simpler. I'm finding ways to export data into Elasticsearch