How managed tables are useful in Madalian Archtecture

[Madalian]

I am having basic question, As managed tables doesnt store their data into ADLS Gen2. But in our architcture we created 3 containers in ADLS Gen2 (Bronze, Silver and Gold) . If I chose managed tables then neither metadata  nor data doesnt store into ADLS Gen2. How these managed tables are used in this type of architecture?

Thanks

[Kaniz_Fatma]

Hi @Madalian , Let’s delve into how managed tables fit into your architecture with Azure Data Lake Storage Gen2 (ADLS Gen2).

1. Managed Tables:

   • Managed tables in Azure Synapse Analytics (formerly known as SQL Data Warehouse) are designed for ease of use and simplified management.
   • They store metadata (such as table schema, column names, data types) but do not directly store data in ADLS Gen2.
   • Instead, managed tables reference external data stored in ADLS Gen2.

2. ADLS Gen2 Architecture:

   • ADLS Gen2 provides a hierarchical namespace, combining the capabilities of Azure Data Lake Storage Gen1 with Azure Blob Storage.
   • It allows you to organize data into directories and nested subdirectories.
   • The hierarchical namespace enables high-performance data access at object storage scale and price¹.

3. Integration:

   • In your architecture, you’ve created three containers: Bronze, Silver, and Gold, within ADLS Gen2.
   • These containers can hold data files (such as Parquet, CSV, or other formats).
   • When you create a managed table, it acts as a logical layer on top of these data files.
   • The managed table’s metadata (schema, column information) is stored in the Azure Synapse Analytics service, not in ADLS Gen2.
   • The managed table references the data files in the appropriate container based on your business logic.

4. Usage:

   • Here’s how managed tables are used in your architecture:
     • Bronze Container: Contains raw, unprocessed data files.
     • Silver Container: Holds semi-processed data (e.g., cleaned, transformed, enriched).
     • Gold Container: Stores fully processed, curated data.
   • You create managed tables in Azure Synapse Analytics, specifying the location (path) of the data files in the corresponding ADLS Gen2 container.
   • Queries against these managed tables retrieve data from the referenced files in ADLS Gen2.
   • The managed table’s metadata simplifies querying and management, abstracting away the underlying file structure.

5. Benefits:

   • Separation of Concerns: Managed tables separate metadata management from data storage, allowing you to focus on query logic.
   • Performance: ADLS Gen2’s hierarchical namespace ensures efficient data access.
   • Security: Leverage Azure RBAC and ACLs to control access to data in ADLS Gen2².

Remember that managed tables provide a convenient layer for querying and managing data, while ADLS Gen2 handles the actual storage. By combining these components, you achieve a scalable and organized architecture for your data processing needs. 🚀