Is databricks capable of housing OLTP and OLAP?

[TinasheChinyati]

Hi data experts.

I currently have an OLTP (Azure SQL DB) that keeps data only for the past 14 days. We use Partition switching to achieve that and have an ETL (Azure data factory) process that feeds the Datawarehouse (Azure Synapse Analytics). My requirement is to migrate the OLTP and OLAP into the lakehouse within databricks. I will have a structured streaming from Eventhub to Bronze then have OLTP as silver and OLAP as Gold. Is this something that is possible within databricks. If so, can you suggest ways to achieve this possibly using delta tables or delta live tables. Thank you

[Kaniz]

Hi @TinasheChinyati, Migrating your OLTP and OLAP workloads into a lakehouse within Databricks is indeed possible. 

Load Data into the Lakehouse:

• Databricks provides tools and capabilities to make data migration to the lakehouse seamless. You can load data from diverse sources into your lakehouse.
• Consider migrating your existing Parquet data lake to Delta Lake, which offers improved reliability, performance, and schema evolution capabilities.
• Explore Lakehouse Federation to query data across different storage layers (Bronze, Silver, Gold) within your lakehouse.

Databricks Data Intelligence Platform vs. Enterprise Data Warehouse:

• The Databricks Data Intelligence Platform is built on top of Apache Spark™, Unity Catalog, and Delta Lake.
• Key differences include:
  • Table-Level Transactions: In Databricks, all transactions are table-level; there are no database-level transactions or locks.
  • Three-Tier Namespacing: Databricks uses a catalog.schema.table pattern, where the terms “database” and “schema” are synonymous.
  • Constraints: Primary key and foreign key constraints are informational only and can only be enforced at a table level.
  • Native Data Types: Be aware that native data types supported in Databricks and Delta Lake might diff....

Delta Live Tables:

• Delta Live Tables is a cloud-native managed service within the Databricks Lakehouse Platform.
• It provides a reliable ETL framework for developing, testing and operationalizing data pipelines at scale.
• By simplifying and modernizing ETL pipelines, Delta Live Tables enables automated intelligent ETL.

In summary, Databricks offers a powerful platform for unifying your data ecosystem, allowing analysts, data scientists, and engineers to work against the same tables. Consider leveraging Delta tables and Delta Live Tables to achieve your lakehouse migration goals. 🚀