Databricks Roadmap Azure
There are a lot of excitement new features coming in 2022. I tried to put them all on one list:
- Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)
- Control metastore, unity creations etc. from admin console,
- CPU clusters in "create cluster" (similar to SQL endpoints) where you specify how much cpus is needed
- Data lineage
- Delta sharing
- Delta: primary keys
- Delta: foreign keys
- Updated partner connect with many new partners joining
- Notebooks UI: transformations wizards (like select, drop, groupby from droplist etc.)
- New global search
- Integration of job service with git
- Jobs monitoring
- Repair and rerun failed tasks and pass values between tasks
- SQL workflows in SQL view
- create table enhanced via wizard
- Delta live tables in GA
- Serverless DBSQL
- Private links (to VNET and to ExpressRoute) and encryption
- ML: LightGBM, Prophet Models and Text Feature in Auto ML
- ML: Feature store integration with Azure MySQL, Azure SQL and CosmosDB
- ML: Feature store: point in time joins, time-based splits
- ML: Feature store API
- ML: Serverless model serving (enable serving button) with high availability SLA
- ML: one click model deployment into registry
In 2023:
- Delta: identity column
- Delta: dynamic checkpoints and incremental commits to limit read/write operations with ADLS
