• Batch (main path): Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables / DLT) is a strong fit for most of your data. Use it for maintenance logs, fuel consumption, and GPS snapshots that arrive in batches. It gives you declarative pipelines, lineage, and built-in data quality. Lakeflow Connect can pull from many sources (databases, SaaS apps, object storage, etc.) into Delta tables that your pipelines then process.
• Real-time: For live telemetry (e.g., live location, engine diagnostics), use Structured Streaming with Lakeflow Connect connectors (Kafka, Event Hubs, Pub/Sub, etc.), typically writing into streaming tables in Lakeflow Spark Declarative Pipelines. Auto Loader can also handle near-real-time ingestion from cloud storage with incremental processing.
• Hybrid: A common pattern is batch for historical and periodic loads, and streaming for critical real-time feeds. Lakeflow Spark Declarative Pipelines supports both in the same pipeline.

Schema design for high-volume time-series

• Liquid clustering: For new tables, use liquid clustering on (vehicle_id, event_timestamp) — this is now the recommended approach over traditional partitioning + ZORDER. It handles data layout automatically and works well with Predictive Optimization, which can manage compaction and clustering for you. If you need partitioning at all, keep it coarse (e.g., event_date or month, and possibly region). If liquid clustering isn't available in your environment yet, fall back to OPTIMIZE ... ZORDER BY (vehicle_id, event_timestamp).
• Layered design: Bronze (raw), Silver (cleaned, deduplicated), Gold (aggregated for dashboards and ML). Keep raw telemetry append-only; do joins and aggregations in Silver/Gold.
• Compaction: For very high volume, liquid clustering plus regular OPTIMIZE keeps file sizes healthy. With Predictive Optimization enabled (default for Unity Catalog managed tables), compaction and vacuum are handled automatically.

• Databricks ML: Use Feature Store (Unity Catalog-integrated) for shared features (vehicle health, route history, maintenance cycles). MLflow tracks experiments and model versions.
• Model Serving: Mosaic AI Model Serving gives serverless deployment with autoscaling and monitoring.
• Patterns: For predictive maintenance, time-series forecasting (Prophet, ARIMA, or gradient boosting) on sensor and maintenance history works well. For ETA, route-based features plus historical trip times are a good starting point.

• AI/BI Dashboards: Native dashboards on top of your Gold tables, good for KPIs and operational views.
• Databricks Apps: For custom UIs (maps, interactive charts, drill-down), you can build apps with Dash (Python) or React + FastAPI. I've used both: Dash for quick internal tools with charts and editable grids, and React + FastAPI for richer apps with caching and connection pooling against the SQL warehouse. Both run on Databricks Apps with OAuth and managed hosting.
• AI/BI Genie Spaces: For natural-language exploration and ad hoc analysis on top of your curated tables.