https://community.databricks.com/t5/technical-blog/tutorial-transform-your-lakeflow-connect-ad-data-into-visual-and/ba-p/166659 <H2><STRONG>Introduction</STRONG></H2> <P><SPAN>Traditionally, unifying requires building brittle custom integrations for every single source. </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/" target="_blank" rel="noopener"><SPAN>Lakeflow Connect</SPAN></A><SPAN> resolves this with native, fully managed connectors for a <A href="https://www.databricks.com/blog/unify-your-marketing-data-lakeflow-connect" target="_blank" rel="noopener">full suite of martech sources</A>, including ad connectors like Google Ads, Meta Ads, and TikTok Ads. Each connector incrementally lands your campaign data directly in </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/" target="_blank" rel="noopener"><SPAN>Unity Catalog</SPAN></A><SPAN>, with no custom API integrations to build or maintain.</SPAN></P> <P><SPAN>In this post, we introduce </SPAN><A href="https://github.com/databricks-industry-solutions/ad-genie" target="_blank" rel="noopener"><SPAN>Ad-Genie</SPAN></A><SPAN>, an open-source accelerator that builds on the raw data ingested by Lakeflow Connect’s ad connectors to deliver three things out of the box: a Spark Declarative Pipeline for transforming data from all sources into one single layer,, an AI/BI dashboard for visual analysis and a Genie agent for conversational analytics. Behind the scenes, Ad-Genie converts the bronze tables ingested by Lakelflow Connect’s Google, Meta, and TikTok Ads connectors into unified, query-ready gold tables. On top of this, Ad-Genie also creates a semantic layer of </SPAN><A href="https://docs.databricks.com/aws/en/uc-semantics/metric-views" target="_blank" rel="noopener"><SPAN>Unity Catalog metric views</SPAN></A><SPAN> that define your KPIs. In just three steps, Ad-Genie delivers an analytics-ready foundation that your team can trust and build on.</SPAN></P> <H2><STRONG>What you’ll build</STRONG></H2> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Giselle_Go_DB_0-1789147689068.png" style="width: 808px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/31021iE3257F048E40830D/image-dimensions/808x293?v=v2" width="808" height="293" role="button" title="Giselle_Go_DB_0-1789147689068.png" alt="Giselle_Go_DB_0-1789147689068.png" /></span></P> <P><SPAN>Ad-Genie transforms raw outputs from Lakeflow Connect’s Meta Ads, TikTok Ads, and Google Ads connectors through the medallion architecture:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Bronze</STRONG><SPAN>: Lakeflow Connect lands raw data from each ad platform into managed tables in Unity Catalog. With fully managed connectors, you can easily set a schedule to incrementally load data from the martech sources of your choice.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Silver</STRONG><SPAN>: Ad-Genie normalizes the bronze layer tables via per-platform source adapters into a common schema using a Spark Declarative Pipeline. For example:</SPAN></LI> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Google's </SPAN><FONT color="#339966"><STRONG><FONT face="courier new,courier">customer_id</FONT></STRONG></FONT><SPAN>, Meta's </SPAN><STRONG><FONT face="courier new,courier" color="#339966">account_id</FONT></STRONG><SPAN>, and TikTok's </SPAN><FONT face="courier new,courier" color="#339966"><STRONG>advertiser_id</STRONG></FONT><SPAN> all become </SPAN><FONT face="courier new,courier" color="#339966"><STRONG>account_id</STRONG></FONT><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Google reports </SPAN><FONT color="#339966"><STRONG>cost_micros</STRONG></FONT><SPAN> (millionths of a currency unit), which is divided by 1,000,000 into a standard spend column.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Platform-specific values like Meta's </SPAN><FONT color="#339966"><STRONG>ARCHIVED</STRONG></FONT><SPAN> or TikTok's </SPAN><FONT color="#339966"><STRONG>DELETE</STRONG></FONT><SPAN> are normalized to a shared set (</SPAN><FONT color="#339966"><STRONG>ACTIVE</STRONG></FONT><SPAN>, </SPAN><FONT color="#339966"><STRONG>PAUSED</STRONG></FONT><SPAN>, </SPAN><FONT color="#339966"><STRONG>REMOVED</STRONG></FONT><SPAN>).</SPAN></LI> </UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Gold</STRONG><SPAN>: Ad-Genie unions the normalized silver tables into cross-platform fact and dimension tables: fact tables carry the metrics (</SPAN><FONT color="#339966"><STRONG>fact_campaign_report</STRONG></FONT><SPAN>, </SPAN><FONT color="#339966"><STRONG>fact_adset_report</STRONG></FONT><SPAN>, </SPAN><FONT color="#339966"><STRONG>fact_ad_report</STRONG></FONT><SPAN>), and dimension tables carry the attributes those metrics are sliced by, like campaign, platform, and date.</SPAN></LI> </UL> <P><SPAN>Beyond the medallion tables, Ad-Genie adds a semantic layer to define your metrics as well as two out-of-the-box interfaces to explore them:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Semantic layer</STRONG><SPAN>: Three Unity Catalog metric views sit on top of the gold tables and define your marketing KPIs. These cover measures like Impressions, Clicks, Spend, ROAS, CTR, and CPA across dimensions like Date, Source Platform, Campaign, and Objective. Both the dashboard and Genie query these views, so the metric logic is defined once and stays consistent.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>AI/BI Dashboard and Genie.</STRONG><SPAN> Ad-Genie optionally deploys two ready-made ways to explore your data: an AI/BI dashboard for visual campaign analysis, and a Genie agent for asking questions in natural language, like "What was our ROAS by campaign last month?" or "Which TikTok ad sets had the highest CPC?"</SPAN></LI> </UL> <H2><STRONG>Getting Started</STRONG></H2> <P><SPAN>Before you begin, make sure you have:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>A Databricks workspace with</SPAN><A href="https://docs.databricks.com/en/data-governance/unity-catalog/index.html" target="_blank" rel="noopener"> <SPAN>Unity Catalog</SPAN></A><SPAN>,</SPAN><A href="https://docs.databricks.com/en/compute/serverless/index.html" target="_blank" rel="noopener"> <SPAN>serverless compute</SPAN></A><SPAN>, and </SPAN><A href="https://docs.databricks.com/aws/en/databricks-ai/partner-powered#for-an-account-disable-or-enable-partner-powered-ai-features" target="_blank" rel="noopener"><SPAN>partner-powered AI features</SPAN></A><SPAN> enabled.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A</SPAN><A href="https://docs.databricks.com/en/compute/sql-warehouse/index.html" target="_blank" rel="noopener"> <SPAN>SQL Warehouse</SPAN></A><SPAN> (required for the dashboard and Genie)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>At least one of the following Lakeflow Connect ad connectors configured and ingesting on a regular schedule: </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/google-ads" target="_blank" rel="noopener"><SPAN>Google Ads</SPAN></A><SPAN>,</SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/meta-ads-overview" target="_blank" rel="noopener"> <SPAN>Meta Ads</SPAN></A><SPAN>, or</SPAN><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/tiktok-ads" target="_blank" rel="noopener"> <SPAN>TikTok Ads</SPAN></A><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/en/dev-tools/cli/index.html" target="_blank" rel="noopener"><SPAN>Databricks CLI</SPAN></A><SPAN> and a bash-compatible shell</SPAN></LI> </UL> <P><SPAN>You can also follow along in this <A href="https://vimeo.com/1222201882/3fa167d4c7" target="_blank" rel="noopener">step-by-step video walkthrough</A>.<BR /><BR /><div class="video-embed-center video-embed"><iframe class="embedly-embed" src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F1222201882%3Fh%3D3fa167d4c7%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F1222201882%2F3fa167d4c7%3Ffl%3Dpl%26fe%3Dvl&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F2195257129-26483f156f5b1924f064e05e0b3815b928d218640c3856ff5471f812a1051009-d_1280%3Fregion%3Dus&amp;type=text%2Fhtml&amp;schema=vimeo" width="600" height="337" scrolling="no" title="Ad Genie" frameborder="0" allow="autoplay; fullscreen; encrypted-media; picture-in-picture" allowfullscreen="true"></iframe></div><BR /></SPAN></P> <P><SPAN>Ad-Genie is packaged as a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/" target="_blank" rel="noopener"><SPAN>Declarative Automation Bundle</SPAN></A><SPAN> and deploys in three steps:</SPAN></P> <H3><STRONG>1. Clone, authenticate, and configure</STRONG></H3> <P><SPAN>Clone the </SPAN><A href="https://github.com/databricks-industry-solutions/ad-genie" target="_blank" rel="noopener"><SPAN>Ad-Genie</SPAN></A><SPAN> repository and authenticate with your Databricks workspace:</SPAN></P> <LI-CODE lang="python">git clone https://github.com/databricks-industry-solutions/ad-genie.git cd ad-genie databricks auth login --host https://&lt;your-workspace-url&gt;</LI-CODE> <P><SPAN>All configuration happens in a single file, </SPAN><FONT color="#339966"><STRONG>config/pipeline.yaml:</STRONG></FONT></P> <LI-CODE lang="python"># What to deploy: full | with_dashboard | with_genie | with_metrics | pipeline_only deployment_profile: full # Required if deploying the dashboard or Genie warehouse_id: abc123def456 # Where Ad-Genie writes silver, gold, and metric-view objects silver_catalog: marketing_analytics silver_schema: silver gold_catalog: marketing_analytics gold_schema: gold mv_catalog: marketing_analytics mv_schema: marketing_semantic_layer # Enable each source and point it to its Lakeflow Connect bronze tables sources: google_ads: enabled: true bronze_catalog: lakeflow_google bronze_schema: google_ads_raw meta_ads: enabled: true bronze_catalog: lakeflow_meta bronze_schema: meta_ads_raw tiktok_ads: enabled: true bronze_catalog: lakeflow_tiktok bronze_schema: tiktok_ads_raw</LI-CODE> <H3><STRONG>2. Deploy</STRONG></H3> <P><SPAN>Run the following commands to validate your </SPAN><FONT color="#339966"><STRONG>config/pipeline.yaml</STRONG></FONT><SPAN> and deploy the Ad-Genie bundle:</SPAN></P> <LI-CODE lang="python">./scripts/sync_config.sh databricks bundle validate -t dev databricks bundle deploy -t dev</LI-CODE> <P><FONT color="#339966"><STRONG>sync_config.sh</STRONG></FONT><SPAN> validates your configuration, updates the bundle definition, and patches the dashboard JSON with your catalog and schema references. </SPAN><FONT color="#339966"><STRONG>bundle deploy</STRONG></FONT><SPAN> then creates your schemas, deploys the pipeline, and registers the dashboard.</SPAN></P> <H3><STRONG>3. Run and explore</STRONG></H3> <P><SPAN>From your Databricks workspace, navigate to </SPAN><STRONG>Workflows &gt; Jobs &amp; Pipelines</STRONG><SPAN>, find </SPAN><SPAN>ad-genie-setup</SPAN><SPAN>, and click </SPAN><STRONG>Run Now</STRONG><SPAN>. This job runs the pipeline to populate your silver and gold tables, creates the three metric views, and provisions the AI/BI dashboard + Genie agent, depending on your deployment profile.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Giselle_Go_DB_1-1789147689068.png" style="width: 815px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/31023i17815E40CF42C241/image-dimensions/815x424?v=v2" width="815" height="424" role="button" title="Giselle_Go_DB_1-1789147689068.png" alt="Giselle_Go_DB_1-1789147689068.png" /></span></P> <P>&nbsp;</P> <P><SPAN>Once the job completes, you can:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Open Catalog Explorer to browse your silver tables (per-source standardized data), your gold tables (unified cross-platform facts and dimensions), and the three metric views.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Open the AI/BI dashboard for a pre-built view of campaign performance across all enabled ad platforms.</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Giselle_Go_DB_2-1789147689069.png" style="width: 817px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/31022iC61BB5806188A54D/image-dimensions/817x429?v=v2" width="817" height="429" role="button" title="Giselle_Go_DB_2-1789147689069.png" alt="Giselle_Go_DB_2-1789147689069.png" /></span></P> <P>&nbsp;</P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Ask the Genie agent questions in natural language, like "What was our total spend by platform last week?" or "Which campaigns had the highest ROAS in Q1?"</SPAN></LI> </UL> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Giselle_Go_DB_3-1789147689069.png" style="width: 819px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/31024i27013136D3C47100/image-dimensions/819x437?v=v2" width="819" height="437" role="button" title="Giselle_Go_DB_3-1789147689069.png" alt="Giselle_Go_DB_3-1789147689069.png" /></span></P> <H2><STRONG>Conclusion</STRONG></H2> <P><SPAN>Together, Lakeflow Connect and Ad-Genie give marketing and data teams a governed, cross-channel analytics layer for their ad platforms—all deployed as a single bundle on Databricks! If you're running campaigns on Google Ads, Meta Ads, or TikTok Ads and want to harness insights from your data, Lakeflow Connect and Ad-Genie are both free to try today:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/ingestion/overview" target="_blank" rel="noopener"><STRONG>Lakeflow Connect</STRONG></A></LI> <LI aria-level="2"><A href="https://www.databricks.com/blog/accelerate-business-insights-lakeflow-connect-now-free-tier" target="_blank" rel="noopener"><SPAN>Lakeflow Connect Free Tier</SPAN></A></LI> </UL> <P><LI-WRAPPER></LI-WRAPPER></P> <P><STRONG>MarTech connector documentation:</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="3"><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/meta-ads" target="_blank" rel="noopener"><SPAN>Meta Ads connector</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="3"><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/tiktok-ads" target="_blank" rel="noopener"><SPAN>TikTok Ads connector</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="3"><A href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/google-ads-concepts" target="_blank" rel="noopener"><SPAN>Google Ads connector</SPAN></A></LI> </UL> <P><STRONG>Ad-Genie</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="2"><SPAN>GitHub: </SPAN><A href="https://github.com/databricks-industry-solutions/ad-genie" target="_blank" rel="noopener"><SPAN>databricks-industry-solutions/ad-genie · GitHub</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Docs: See the </SPAN><SPAN>docs/</SPAN><SPAN> directory in the repository for configuration, deployment, and extension guides</SPAN></LI> </UL> <P><SPAN>Since Lakeflow Connect is launching new ingestion connectors rapidly, Ad-Genie is designed to grow with your stack. Adding a new ad platform is straightforward: create a source adapter that maps the platform's schema to the standardized format, register it, enable it in your config, and redeploy! For questions or contributions, open an issue on GitHub or reach out to your Databricks account team.</SPAN></P> <P><SPAN>Take advantage of the </SPAN><A href="https://www.databricks.com/blog/accelerate-business-insights-lakeflow-connect-now-free-tier" target="_blank" rel="noopener"><SPAN>Lakeflow Connect Free Tier</SPAN></A><SPAN> and deploy </SPAN><A href="https://github.com/databricks-industry-solutions/ad-genie" target="_blank" rel="noopener"><SPAN>Ad-Genie</SPAN></A><SPAN> today. Within an afternoon, you can have a governed dashboard and a Genie agent that your marketing team can use to optimize ad spend!</SPAN></P> Fri, 11 Sep 2026 19:04:51 GMT abhishek-iyer 2026-09-11T19:04:51Z https://community.databricks.com/t5/technical-blog/tutorial-transform-your-lakeflow-connect-ad-data-into-visual-and/ba-p/166659 <P>Learn how the Ad-Genie accelerator deploys an AI/BI dashboard and a Genie agent to harness data ingested by Lakeflow Connect’s Google Ads, Meta Ads, and TikTok Ads connectors.</P> Fri, 11 Sep 2026 19:04:51 GMT https://community.databricks.com/t5/technical-blog/tutorial-transform-your-lakeflow-connect-ad-data-into-visual-and/ba-p/166659 abhishek-iyer 2026-09-11T19:04:51Z