Tutorial: Using Databricks Apps as a Lambda with Zerobus to write to Delta

[Vicky_Bukta_DB]

There are multiple ways to “push” data to your lakehouse. Traditional ingestion methods, such as batch jobs, staging layers and complex pipelines can slow down time to insights and add increase operational overhead. This complexity becomes more challenging with event data or high-velocity telemetry that demands near real-time processing.

Zerobus Ingest, part of Lakeflow Connect, lets you push event data directly to your lakehouse with high-throughput, and low-latency from source to Delta tables.  It removes unnecessary data hops, reduces operational overhead, and integrates seamlessly with advanced analytics and AI tools 

This Databricks Direct Write App offers a hands-on way to experience the impact of Zerobus Ingest. Built as a lightweight FastAPI application on Databricks Apps, it generates sample telemetry data and lets you compare ingestion methods side-by-side. 

In this blog post, we’ll demonstrate how  to build the “Databricks Direct Write App,” and how it integrates with Zerobus Ingest. Within the app, you will be able to toggle between different ingestion mechanisms and find the best one for your needs. Check it out!

🎯 1. Application Summary

Key Value Proposition:

The app solves the challenge of slow or complex data ingestion by providing an isolated, web-based interface that leverages high-speed, direct API connections (like Zerobus Ingest) to write data instantly to Delta Lake, bypassing traditional data loading paths.

Core Features:

• Multi-Writer Architecture: Lakeflow Connect supports the Zerobus Ingest (high-performance streaming), Direct JDBC SQL (SQL-based writing via Databricks SDK).
• High Performance: Achieved primarily through the Zerobus Ingest Direct Write API and Protobuf serialization.
• Security: Uses Platform-Managed Service Principal authentication, eliminating the need to store credentials manually.
• Observability: Includes comprehensive structured logging, detailed performance metrics (processing time, throughput), and debug endpoints.

🏗️ 2. Architecture Layout and Description

The application follows a clear, three-tier architecture, implemented as a single FastAPI application deployed on the Databricks Apps platform.

Architecture Diagram:

Key Components:

1. Web Interface (static/index.html): A responsive, interactive front-end that allows users to select a writer, input product data, and view real-time status and performance metrics.
2. FastAPI Application (main.py): The core server that handles routing (/api/v1/process-structured), authentication, logging, and dynamic writer selection. It uses Pydantic Models for rigorous incoming data validation.
3. Writer System (writers/): A modular system implementing the DataWriterInterface. It contains the logic for different ingestion methods, instantiated via a factory based on the user's selection.
4. Delta Tables: The final destination for the ingested data in the Databricks Unity Catalog.

🚀 3. Focus on Zerobus Ingest Direct API Write Method

The Zerobus Writer (writers/zerobus.py) is the high-performance implementation designed for fast, streaming ingestion.

Zerobus Ingest Writer Features:

• Direct Write: Utilizes the Zerobus Direct Write API for high-throughput data transfer.
• Serialization: Uses Protobuf (product_record.proto) for efficient, compact data serialization before transmission.
• Async Streaming: Implements asynchronous logic for optimal non-blocking performance.
• Security & Authentication: Automatically detects and utilizes OAuth2 Service Principal credentials injected by the Databricks Apps environment. 
• Error Handling: Robust connection and stream error handling with comprehensive logging.

🛠️ 4. Step-by-Step Tutorial (Zerobus Ingest Write)

This tutorial covers the prerequisite setup and the steps to use the Zerobus Ingest Writer via the application interface.

Step 1: Databricks Setup and Permissions (Prerequisites)

1. Create Service Principal: Create a dedicated Service Principal in your Databricks Workspace (e.g., 'zerobus-writer').
   CREATE SERVICE PRINCIPAL 'zerobus-writer';
   
   
2. Grant Permissions: Grant the Service Principal the necessary permissions on the target Delta Table (kaustavpaul_demo.zerobus_delta.zerobus_products_clean).
   GRANT USE_CATALOG ON CATALOG kaustavpaul_demo TO `zerobus-writer`;
   GRANT USE_SCHEMA ON SCHEMA kaustavpaul_demo.zerobus_delta TO `zerobus-writer`;
   GRANT MODIFY, SELECT ON TABLE kaustavpaul_demo.zerobus_delta.zerobus_products_clean TO `zerobus-writer`;
   
   

Step 2: Deployment and Configuration

1. Configure databricks.yml: Specify the Service Principal name in your databricks.yml file under the app permissions.
   resources:
     apps:
       databricks-delta-app:
         # ... other configuration
         permissions:
           - service_principal_name: "zerobus-writer"
             level: CAN_MANAGE
   
   
2. Deploy the Application: Deploy the application bundle using the Databricks CLI.
   databricks apps deploy
   
   Note: Databricks Apps automatically handles injecting the Service Principal's credentials (Client ID/Secret) as environment variables.

Step 3: Web Interface Usage (Testing the Zerobus Ingest Write)

1. Access the Application: Navigate to the URL provided after deployment.
2. Select Writer: From the dropdown menu on the web interface, select "Zerobus Writer."
3. Input Data: Fill in the structured product information (e.g., Product ID: PROD005, Product Name: Smart Watch, Price: 499.99, Category: electronics).
4. Submit: Click "Process Products Data."
5. Review Status: The application will display a real-time status update, including:

• Success/Failure message.
• Performance metrics (Processing time, Throughput).
• Confirmation that the Zerobus Writer was used.

Step 4: Verification

1. Check Debug Endpoint: For API-level verification, use the debug endpoint:
   curl https://your-app-url/debug/zerobus-availability
   
   Expected result will show zerobus_writer_available: true.
2. Check Delta Table: Run a SELECT query in your Databricks SQL endpoint to confirm the new record has been successfully streamed to the target Delta table by the Service Principal.

Github Link: https://github.com/kaustavpaul107355/databricks-starter/tree/main/databricks-starter/databricks-apps... 

More in the Zerobus Ingest series: Once you’ve explored the Direct Write App workflow, see how the same high-performance Zerobus APIs can power end-to-end ingestion in other environments:

• Zerobus Ingest Station – Wrap the Zerobus SDK in a custom FastAPI app to expose secure, schema-validated REST endpoints for your partners and devices.
• Zerobus Ingest RabbitMQ Forwarder – Deploy lightweight agents that subscribe to message queues like RabbitMQ and stream data directly into Delta tables.