We are in the same situation. We have a CDH cluster with IaaS architecture. The data are on Hdfs in EC2 disks in AWS and we want to migrate the data from CDH to Databricks in AZURE.

If we federate CDH's HIVE metastore with Databricks, we can migrate the data very fast with incremental queries on SparkSQL on Databricks. What do you think? Is it possible?