Rjdudley
Honored Contributor

interesting to know how will the strategy differ incase of incremental loads. 

There are three data "sites" you need to take into account--your on-prem, your AWS, and Databricks AWS.  Under normal circumstances it is significantly easier to push data from on-prem into the cloud.

If you used HDFS you don't have a lot of options, and on-prem adds a layer of difficulty.  Presumably your InfoSec and networking teams have a lot of protections in place to prevent something from outside your network connecting and grabbing all your data.  It's not impossible, it's just a larger discussion you need to have internally.

For the actual data migration, there are a number of Apache tools with Hadoop integration you can use to build something, and AWS has their versions of some of those, but you'll still need to get the networking figured out.  If you want a product recommendation, CData has one which can sync from HDFS to Databricks: HDFS Integrations: Drivers & Connectors for HDFS.  No experience with this one but overall their stuff is really good.

Adding further complexity is Databricks' serverless compute.  That runs in Databricks' AWS account, not your AWS account.  That's another set of networking permissions.  Databricks classic compute runs in your AWS environment and might be a simpler option depending on what you decide to do.

One question I would ask is, how much of the Cloudera data lake do we want to replicate in the Databricks lakehouse?  If the answer is not much, I would consider loading only the raw layer and processing everything in Databricks.

> So in case of Databricks express set up , do we still need to have AWS S3 bucket for storage. 

Oh yes, and in fact you'll have many S3 buckets.  You may have buckets for landing, and you'll definitely have buckets for data in bronze, silver and gold.  I recommend using one bucket in bronze for each source.  How you organize silver and gold is up to your data governance.

Express setup doesn't build your AWS cloud infrastructure, only sets up your Databricks account and a metastore.  Databricks is a large SaaS platform with a lot of moving pieces, it's not a single-instance database like anything RDS.  Take the time to go through the Lakehouse Fundamentals course.