Data+AI Summit 2024 Data Warehousing Announcements

lara_rachidi — Tue, 23 Jul 2024 09:51:08 GMT

In a nutshell

AI/BI Dashboards (GA) allow end users to create dashboards with low-code experience via a drag-and-drop canvas or through natural language via AI-powered text to viz capability. → Data+AI Summit Talk
AI/BI Genie (Public Preview) is a conversational experience for business users to interrogate their data through natural language and follow up with visualisations. You can tune and improve responses to produce accurate and reproducible answers. → Blog Post
AI functions (Public Preview) allow you to invoke Gen AI models to perform specific tasks such as sentiment analysis, classification, summarisation, translation, etc. on columns in structured tables using SQL functions. You can wrap any custom model serving endpoint in a SQL function. → Data+AI Summit Talk

Databricks AI/BI is the new type of business intelligence product. Built on compound AI systems that combine interacting components such as understanding data and comments, creating complex SQL queries as well as visual generation, AI/BI empowers business users to carry out self service analytics. There are 2 complementary products:
AI/BI Dashboards (formerly Lakeview dashboards) is GA. Dashboards allow end users to create dashboards with low-code experience via a drag and drop canvas or through natural language via AI powered text to viz capability. Dashboards offer a wide set of visualisation capabilities that includes cross filtering, as well as sharing and exporting options.

AI/BI Genie is now in Public Preview. AI/BI Genie is the conversational experience for business users to interrogate their data through natural language and follow up with visualisations with a single click. Genie uses a set of capabilities to allow teams to tune and improve responses so that it produces trustworthy & accurate answers that are reproducible:
Setting up the space with pre set questions to get business users started
Asking for clarification with follow ups if it does not have sufficient information and uses the response to guide prompts.
Unity Catalog table and column comments to enrich Genie’s knowledge base so that it understands an organisation’s terminology and business jargon
Instructions that are saved as text to guide the LLMs behaviour that would be reflected in the prompt
Sample SQL statements that users can save to teach the model to answer specific questions
Certified answers that can be saved to answer specific questions so that results are always reproducible with an indicator to show end users that the answers are trustworthy
Logging end user’s questions and answers for quality monitoring

Watch the Data+AI summit talk here about AI/BI and Genie with a demo.
Learn more about the AIBI experience with Miranda Luna, Sr Manager Product Management and Chao Cai, Sr Director of Engineering.

Data warehousing performance improvements: watch the Data+AI summit talk with a demo here.
Automatic Liquid Clustering in private preview. Liquid clustering allows you to optimise data layouts for performance by specifying cluster keys, which can evolve over time without rewriting existing data. Auto Liquid Clustering uses AI to learn from the query patterns on your workloads and automatically selects and apply the cluster keys that will bring optimal performance.

Auto Statistics in private preview uses a model to return columns that have higher fidelity data to be collected automatically on writes for query optimiser to generate an optimal plan to improve Spark execution. This removes the need for explicit user action with running ANALYZE to generate query optimizer stats.

Predictive I/O 2.0 is an improvement to Predictive I/O (in GA), which uses deep learning to improve query access patterns by estimating the next matching row, optimising the data scanned. 2.0 now has larger AI models powered by Mosaic that work with a larger set of feature vectors, which can now accelerate much more workloads.

AI Functions in public preview. AI functions allows you to invoke Gen AI models and apply AI to perform specific tasks such as sentiment analysis, classification, summarisation, extraction, translation on columns in structured tables using SQL functions. AI Query extends this capability beyond pre-canned functions by allowing you to wrap any custom model serving endpoint in a SQL function.

Subscribe to the NextGenLakehouse to receive monthly updates!