Some key notes from today's Databricks Azure Roadmap Q4 event:
- Faster merge and updates with Photon + Deletion vectors in Q3,
- Unity Catalog - system tables, including system table for lineage,
- ML features and models under Unity Catalog governance,
- Auto tuning for managed tables in Unity Catalog
- Delta sharing Now GA,
- Databricks marketplace (private preview in Q4),
- The private link between Databricks Control Plane and corporate network,
- HIPAA and other compliance GA,
- Customer managed keys from databricks services and managed disks coming soon,
- Cost management in databricks console, charts, budgets, etc. (in development),
- Serverless SQL (public preview),
- Materialized views (in development),
- Automatic workload optimization (so small queries don't need to wait for big ones to finish),
- Full REST API, including SQL querying (private preview)
- Create an external table in any database of your choice and query it from databricks,
- Pythons UDFS so your python code can be registered directly as a permanent function in the catalog,
- SQL queries in workflows,
- Run SQL query as the service account,
- Dashboards in git,
- Enhanced workflow logic, including True/Falce, Fail/Success steps,
- Workflow trigger when new data arrive (private preview)
- ML autoscaling endpoint to serve ML models in production (private preview)
- Delta live tables integrated with ML,
- The feature store can be combined with Delta Live Tables (private preview)
- MLOps templates (private preview)
- Share dashboards to non-databricks users,
- Include dashboards on the web page,
- The e-nzyme in development (optimal update techniques)
- SCT Type 2 in preview,
- Serverless jobs (private preview in Q3),
- Enhanced monitoring and alerts, including visuals and Slack and other hooks,
- Primary/foreign keys in public preview,
- Jupiter support in notebooks,
- Repos enhancements (rebase etc.)
- Workspace browser to see all objects (aka file browser) (in development)
- VS code extensions Private Preview Q3
