I have an existing data pipeline which looks like this: A small MySQL data source (around 250 GB) and data passes through Debezium/ Kafka / a custom data redactor -> to Glue ETL jobs and finally lands on Redshift, but the scale of the data is too small. I am trying to build a new data pipeline from the following current state: - 3 MySQL replicas (source) in 1 cluster (total data size 1 TB) - And data scientists directly query the data in a hacky way resulting in hours of delay. - But for this new pipeline, the data is much larger in TBs. So existing pipeline implementation fails I am brainstorming architectural ideas at this point. A lot of these concepts are quite new to me. So, have some questions which are unclear at this point: 1. Is it possible to create a CDC pipeline inside the SQL cluster? Is DeltaLakes a good use case with MYSQL replicas? 2. Is there a good way to redact/hide PII information inside the SQL cluster? I was thinking of creating views that don't read the PII fields. What is an alternative to Debezium/Kafka if I am using Databricks and want to hide PII fields upstream? 3. I am looking at AWS Databricks as an option to perform ETL operations. But in this case, should the source be the CDC tables before ETL starts? Or is it advisable to build another CDC pipeline followed by another one to read from the output of this pipeline? (This portion is quite unclear to me) 4. What would be good data ingestion options in this scenario? I am thinking of a pull-based architecture where Spark reads the data source in batch intervals. Is there a better approach? 5. Do I really need Redshift if the data source is just 1 TB? Would a DW be overkill at this point? Can I not just create structured tables in DataBricks itself?