https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/ba-p/148399 <P><SPAN><A href="https://docs.databricks.com/aws/en/ingestion/zerobus-overview" target="_blank" rel="noopener">Zerobus Ingest</A> now supports Databricks Variant type via <A href="https://docs.databricks.com/aws/en/ingestion/zerobus-ingest" target="_self">REST API</A> (Beta), enabling schema-free JSON ingestion. No more schema definitions, no more ETL headaches—just send your data and query it.</SPAN></P> <H2><SPAN>The Schema Management Problem</SPAN></H2> <P><SPAN>If you've worked with data ingestion pipelines, you know the drill: define a schema, update it when your data structure changes, redeploy your pipeline, and hope nothing breaks. This cycle becomes especially painful when ingesting semi-structured data from APIs, logs, or IoT devices, where data schema evolution is frequent.</SPAN></P> <P><SPAN>Traditional approaches force you to choose between:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Rigid schemas</STRONG><SPAN> that break when data structures change</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>String/JSON columns</STRONG><SPAN> that sacrifice query performance and type safety</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Complex ETL</STRONG><SPAN> to normalize everything into fixed tables</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Variant-type-blog-post.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/24623iAA0B67EB9C3A1831/image-size/large?v=v2&amp;px=999" role="button" title="Variant-type-blog-post.png" alt="Variant-type-blog-post.png" /></span></P> <H2><SPAN><BR />Enter Variant Type Support in Zerobus Ingest</SPAN></H2> <P><SPAN>Databricks Variant type provides a native way to store and query semi-structured data without predefined schemas. With Zerobus Ingest's REST API support for Variant, you can now ingest JSON directly while maintaining query-performant reads.</SPAN></P> <H2><SPAN>Three Key Benefits</SPAN></H2> <OL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Limited Schema Definitions Required: </STRONG><SPAN>Send your JSON data as-is. Add new fields tomorrow without updating schemas or redeploying pipelines.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Native Performance</STRONG><SPAN>: Unlike storing JSON as strings, Variant types are stored in an optimized format, where “Predictive Optimization” can apply </SPAN><A href="https://docs.databricks.com/aws/en/delta/variant-shredding" target="_blank" rel="noopener"><SPAN>shredding</SPAN></A><SPAN> to enhance read performance.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Simplified Pipelines</STRONG><SPAN>: Reduce your ETL code significantly. No more:</SPAN></LI> <OL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Manual schema inference logic</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>JSON parsing and flattening steps</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Schema evolution tracking</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Data type conversion errors</SPAN></LI> </OL> </OL> <P><SPAN>The Zerobus Ingest REST API endpoint becomes your ingestion pipeline. Focus on business logic instead of data plumbing.</SPAN></P> <H2><SPAN>Getting Started</SPAN></H2> <P><SPAN>To enable Variant support in your Zerobus Ingest workflow:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Create a target table with a Variant column</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Start ingesting JSON data without schema definitions</SPAN></LI> </OL> <H2><SPAN>Step 1: Creating Your Target Table</SPAN></H2> <P><SPAN>Before ingesting data, create a target table with a Variant column. Zerobus Ingest requires the table to exist beforehand—it will not auto-create tables for you.</SPAN></P> <P><SPAN>Here's a simple table definition:</SPAN></P> <LI-CODE lang="python">CREATE TABLE main.default.events ( event_id STRING, data VARIANT, ingested_at TIMESTAMP );</LI-CODE> <P><SPAN>This minimal schema gives you:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>An identifier field for tracking events.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A </SPAN><SPAN>VARIANT</SPAN><SPAN> column to store your entire JSON payload.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A timestamp for ingestion tracking.</SPAN></LI> </UL> <P><SPAN>You can also use a single-column approach if you don't need additional metadata:</SPAN></P> <LI-CODE lang="python">CREATE TABLE main.default.events ( data VARIANT );</LI-CODE> <P><SPAN>The beauty of Variant is that all your JSON structure lives in that one column, queryable without further schema definitions.</SPAN></P> <H2><SPAN>Step 2: Ingest Data</SPAN></H2> <P><SPAN>Using the Zerobus Ingest REST API with Variant is straightforward. Here's a simple example:</SPAN></P> <LI-CODE lang="markup">curl -X POST https://&lt;databricks-workspace&gt;.cloud.databricks.com/zerobus/v1/tables/main.default.events/insert \ -H "Authorization: Bearer &lt;token&gt;" \ -H "Content-Type: application/json" \ -d '[ { "event_id": "evt_123", "data": { "id": "usr_456", "email": "user@example.com", "preferences": { "notifications": true, "theme": "dark" } }, "timestamp": "2026-02-13T10:30:00Z", }, ... ]'</LI-CODE> <P><SPAN>Once ingested, query your data with standard SQL and </SPAN><A href="https://docs.databricks.com/aws/en/semi-structured/variant#sql-shorthand-for-variant_get" target="_blank" rel="noopener"><SPAN>Variant shorthand SQL</SPAN></A><SPAN>:</SPAN></P> <LI-CODE lang="python">SELECT event_id, data:id as user_id, data:preferences.theme as theme, data:email as email FROM events WHERE data:preferences.notifications = true;</LI-CODE> <H2><SPAN>Real-World Use Cases</SPAN></H2> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>API Data Lakes</STRONG><SPAN>: Ingest webhook payloads without mapping every possible field upfront.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Log Aggregation</STRONG><SPAN>: Collect application logs with varying structures from different services in a single table.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>IoT Telemetry</STRONG><SPAN>: Store sensor data where device capabilities and metrics evolve over time.</SPAN></LI> </UL> <H2><SPAN>Conclusion</SPAN></H2> <P><SPAN>Zerobus Ingest with Variant type support removes the friction from semi-structured data ingestion. By combining the flexibility of schema-free JSON with the performance of native Databricks storage, you can build more resilient data pipelines with less code and less maintenance for maximum flexibility.</SPAN></P> <P><SPAN>Ready to simplify your data ingestion? Check out the </SPAN><A href="https://docs.databricks.com/aws/en/ingestion/zerobus-overview" target="_blank" rel="noopener"><SPAN>Zerobus Ingest documentation</SPAN></A><SPAN> and start sending data today.</SPAN></P> <P><I><SPAN>Have questions or want to share your Variant use cases? Join the discussion below!</SPAN></I></P> Fri, 06 Mar 2026 23:11:29 GMT Vicky_Bukta_DB 2026-03-06T23:11:29Z https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/ba-p/148399 <P><SPAN>No more schema definitions, no more ETL headaches—just send your data and query it.</SPAN></P> Fri, 06 Mar 2026 23:11:29 GMT https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/ba-p/148399 Vicky_Bukta_DB 2026-03-06T23:11:29Z https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/bc-p/149102#M929 <P>This really a great feature!<BR />Having that option years ago would have made my life so much easier.<BR />Thank for sharing.</P><P>I see from the <A href="https://docs.databricks.com/aws/en/delta/variant-shredding" target="_self">documentation</A>:</P><P class="lia-indent-padding-left-30px"><SPAN>Shredding improves the query performance of&nbsp;</SPAN>VARIANT<SPAN>&nbsp;data by storing commonly occurring fields as separate columns in the Parquet files. This process reduces the I/O required to read fields and improves compression using a columnar format instead of a binary blob.</SPAN></P><P><SPAN>Is there any<STRONG> known limit</STRONG> on how many fields are shreded? Also, what is the logic that define '<STRONG>commonly occurring fields'</STRONG>?</SPAN></P> Mon, 23 Feb 2026 20:05:08 GMT https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/bc-p/149102#M929 wesleyfelipe 2026-02-23T20:05:08Z https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/bc-p/149490#M942 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/209480">@wesleyfelipe</a>&nbsp;- thanks for the comment.</P><P>We don't publicly state the max field limit because it's subject to change, but think about 100+.</P><P>The *commonly occurring fields* decision is a heuristic based on write sampling, historical write data, and query history. Ie, whether the given value is common in a current batch, whether it is historically common for a given column, and whether it is frequently used in filters.</P> Fri, 27 Feb 2026 10:06:08 GMT https://community.databricks.com/t5/technical-blog/schema-free-data-ingestion-with-zerobus-and-variant-type/bc-p/149490#M942 atomic 2026-02-27T10:06:08Z