Note: This blog post is a summary of a Databricks Genie step-by-step tutorial. The full tutorial and code are available for free on GitHub.
Tutorial: Databricks Genie for Data Engineers and Data Scientists
“Take a quick look at the data and assess how useful it could be for the business.”
It is your first week on the team. The request sounds simple. Then you open the table: 696 million flight telemetry records from 54,093 aircraft. Now you need to understand what the records contain and which data you can trust.
- Where do the measurements look wrong?
- Which business questions can the data answer?
- How do I start?
Before you build a pipeline and workflows, you need enough evidence to make those decisions.
This tutorial uses Databricks Genie to explore OpenSky flight telemetry data, publicly available through OpenSharing, to define data quality rules, and to build a pipeline and an app. It is written for data engineers and data scientists.
- Genie Agents generate SQL, query results, and charts from questions in plain English.
- Genie Code generates pipelines, workflows, and application code. Both run 100% serverless on Databricks Free Edition.
Setup instructions, prompts, and code are available in GitHub: AI-powered Analytics of 700 Million OpenSky Records. Try it in Databricks Free Edition!
What this Databricks Genie tutorial works with
- Dataset: 696 million state vector records from The OpenSky Network, 54,093 aircraft, 1 March 2026.
- Access: read-only Unity Catalog table marketplace.opensky.state_vectors through OpenSharing, no copy of the source data.
- Real data: genuine outliers and missing values, not a pre-cleaned sample.
- Scope: access, profiling, exploration, pipeline, job, app, and reading the result from outside Databricks.
- AI-tooling focus: learn to use Genie, not write pipeline code
The workshop content at a glance
1. How do you access OpenSky Network data through Databricks Marketplace?
You subscribe to the OpenSky listing in Databricks Marketplace. Roughly 696 million flight records for 1 March 2026 arrive through OpenSharing as a read-only Unity Catalog table, without copying the source data into your account.
Outcome: direct access to the marketplace.opensky.state_vectors table via OpenSharing.
2. How do you find data quality issues in OpenSky data with Genie Agents and Genie Code?
Genie Agents and Genie Code run the exploratory data analysis on the 696 million OpenSky records. They profile columns, evaluate range limits, and flag anomalies, and they write the SQL that does it.
Outcome: a data profile summarizing 67 quality validations across missing, out-of-range, and implausible measurements.
3. How do you explore and visualize OpenSky data with Genie Agents?
You ask your Genie Agents questions in plain English about aircraft positions, speeds, and flight behavior. Genie writes the query, runs it, and picks the visualization.
Outcome: interactive maps of aircraft positions and distribution plots, generated directly from natural language prompts.
4. How do you build a data quality pipeline and a workflow with Genie Code?
Genie Code converts the 67 OpenSky data quality findings into an Apache Spark™ Declarative Pipelines (SDP) workflow structured in Bronze, Silver, and Gold layers. You declare streaming tables and materialized views with data quality constraints, and the runtime handles orchestration and incremental processing.
Outcome: a medallion pipeline, wrapped in a multi-task workflow, with reviewed data quality expectations and regional analytics tables for Americas, EMEA, and APAC.
5. How do you build a Databricks App for flight analytics with Genie Code?
Genie Code generates an interactive Databricks App for investigating flight trajectories and holding patterns in the OpenSky data.
Outcome: a Databricks App that visualizes flight trajectories in a space-time prism, with no manual SQL, no web development, and no hosting to manage.
6. How do you read OpenSharing tables with Python outside Databricks?
You publish the cleaned OpenSky tables through OpenSharing and query them locally with Python, pandas, and VS Code. No Spark and no Java involved.
Outcome: a lightweight Python workflow for reading shared tables externally.
What you can take back to the team
You now have an OpenSky data profile, a pipeline with enforced quality expectations, regional analytics tables, and an app for exploring flight trajectories. You can show the team what the records contain, explain which measurements the pipeline excluded, and demonstrate how to investigate a possible holding pattern.
Genie wrote the SQL, the pipeline, and the code. It handles a lot of the mechanical, repetitive work, but that doesn't make the human obsolete. It didn't decide where a broken sensor ends and an unusual flight begins, that threshold is still a judgement call. And it's still the user deciding what permissions Genie runs with.
How to run the Genie for Data Engineers tutorial?
If this overview was useful, get hands-on with the full tutorial, AI-powered Analytics of 700 Million OpenSky Records, on Databricks Free Edition:
- Access OpenSky data through Marketplace
- Profile data and investigate anomalies with Genie
- Explore and visualize with Genie Agents
- Build a pipeline with Genie Code
- Create a Lakeflow Job with Genie Code
- Build a Databricks App with Genie Code
- Read OpenSharing tables with Python
