For anyone who hasn't tried it yet: Designer is a visual, AI-native canvas for pipeline authoring that runs inside the workspace. Every visual operation compiles to a real Lakeflow Spark Declarative Pipeline, files are saved as .designer.ipynb in Git Folders, and governance is inherited from Unity Catalog rather than reconfigured.
The five approaches
1. Drag-and-drop. Place operators on the canvas and wire them left to right. The palette covers sources, filters, joins, aggregates, pivots, AI functions, SQL, Python, charts and notes. Best when you want precise control over each step.
2. Single prompt. Describe the whole pipeline to Genie Code in one specification and let it scaffold the graph. I used one prompt to build a bronze/silver/gold cohort pipeline with streaming ingestion, casting, liquid clustering and CLV segmentation.
3. Multi-prompt refinement. Build conversationally, one node per turn, checking the data preview as you go. Thirteen prompts produced a sales performance and forecasting pipeline. Each turn extended the graph rather than regenerating it.
4. From externally ingested sources. Designer picks up from tables landed by managed ingestion, so the connector work happens upstream and the canvas starts at bronze.
5. Input-to-output synthesis. Define your sources and your target shape โ you can even attach a screenshot of the desired output โ and let Genie Code infer the joins and aggregations between them.
The thing that surprised me
Node count fell steadily as more of the design was delegated:
- Drag-and-drop โ 22 nodes
- Multi-prompt โ 16 nodes
- Single prompt โ 13 nodes
- External sources โ 11 nodes
- Input-to-output โ 8 nodes
That isn't a quality ranking. A granular graph is easier to debug node by node; a compact one is easier to read at a glance. But it's a useful way to pick an approach: single-prompt when you're implementing a design you already have, multi-prompt when you're still working out what the design should be.
Three practical notes
Prompt specificity determines output quality. Fully-qualified table names, explicit casts, explicit clustering. Vague prompts produce vague pipelines โ that was the single biggest factor in how usable the generated graph was.
Use the 1,000-row preview while building. Every node carries a data preview, persisted per node. Build against the sample and switch to the full dataset only for final validation.
Review generated code like a submission. Genie Code is grounded in Unity Catalog metadata and it's fast, but a generated CASE expression deserves the same scrutiny as a hand-written one. Check your boundary conditions โ BETWEEN 500 AND 1000 and > 1000 need to agree on 1000.
Happy to answer questions on any of the five approaches. If anyone has pushed Designer further โ particularly on user-defined operators or CI/CD promotion through Declarative Automation Bundles โ I'd be interested to hear how that went.
Full walkthrough, with canvas screenshots for each approach, the complete prompts I used, and the Git/CI-CD and UDO sections: Beyond Visual ETL: How Databricks Lakeflow Designer Unifies Pro-Code Power and AI-Native No-Code Sim...
