Databricks Roadmap Azure

There are a lot of excitement new features coming in 2022. I tried to put them all on one list:

• Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)
• Control metastore, unity creations etc. from admin console,
• CPU clusters in "create cluster" (similar to SQL endpoints) where you specify how much cpus is needed
• Data lineage
• Delta sharing
• Delta: primary keys
• Delta: foreign keys
• Updated partner connect with many new partners joining
• Notebooks UI: transformations wizards (like select, drop, groupby from droplist etc.)
• New global search
• Integration of job service with git
• Jobs monitoring
• Repair and rerun failed tasks and pass values between tasks
• SQL workflows in SQL view
• create table enhanced via wizard
• Delta live tables in GA
• Serverless DBSQL
• Private links (to VNET and to ExpressRoute) and encryption
• ML: LightGBM, Prophet Models and Text Feature in Auto ML
• ML: Feature store integration with Azure MySQL, Azure SQL and CosmosDB
• ML: Feature store: point in time joins, time-based splits
• ML: Feature store API
• ML: Serverless model serving (enable serving button) with high availability SLA
• ML: one click model deployment into registry

In 2023:

• Delta: identity column
• Delta: dynamic checkpoints and incremental commits to limit read/write operations with ADLS