topic Re: Wap pattern unity catalog in Data Engineering

Wap pattern unity catalog

jorperort — Mon, 30 Dec 2024 16:30:24 GMT

Good afternoon,

I am looking for documentation to implement the WAP pattern using Unity Catalog, workflows, SQL notebooks, and any other services necessary to use this pattern. Could you share information on how to approach the problem with documentation, a practical case, or any other pattern or best practices I should consider? Perhaps separating a staging schema, for example.

Best regards, and thank you in advance for your help.

Re: Wap pattern unity catalog

VZLA — Mon, 30 Dec 2024 16:35:43 GMT

@jorperort thanks for your question!

To implement the Write-Audit-Publish (WAP) pattern in Databricks using Unity Catalog, workflows, and SQL notebooks, follow these steps:

Set Up Unity Catalog: Configure Unity Catalog for unified governance across data assets.
Create Schemas: Use separate schemas for staging and production to manage data lifecycle:
- CREATE SCHEMA IF NOT EXISTS staging;
- CREATE SCHEMA IF NOT EXISTS production;
Develop SQL Notebooks: Write SQL notebooks for ingestion, validation, and transformation tasks:
- Ingest data into staging.
- Validate and transform the data.
- Publish to production.
Automate with Workflows: Set up Databricks Workflows to automate notebook execution in sequence: ingest, validate, transform, and publish.
Follow WAP Steps:
- Write: Load raw data into the staging schema.
- Audit: Validate and transform data within staging.
- Publish: Move validated data to production.

Re: Wap pattern unity catalog

szymon_dybczak — Mon, 30 Dec 2024 16:45:11 GMT

Hi @jorperort ,

Apart from nice step by step instruction that @VZLA has provided, you can also take a look at short presentation of WAP pattern at the official databricks YT channel:

https://youtu.be/4K3zAmUgViE?t=492