topic Data+AI Summit 2024 Data Engineering Announcements in Announcements

topic Data+AI Summit 2024 Data Engineering Announcements in Announcements https://community.databricks.com/t5/announcements/data-ai-summit-2024-data-engineering-announcements/m-p/78383#M220 <H1 id="ae1b" class="lz ma fp be mb mc md me mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw bj" data-selectable-paragraph="">In a nutshell</H1> <UL class=""> <LI id="1918" class="la lb fp lc b ld mx lf lg lh my lj lk ll mz ln lo lp na lr ls lt nb lv lw lx nc nd ne bj" data-selectable-paragraph=""><STRONG class="lc fq">Databricks</STRONG><SPAN> </SPAN>acquired<SPAN> </SPAN><STRONG class="lc fq">Tabular</STRONG>, a data management company founded by Ryan Blue, Daniel Weeks, and Jason Reid. This acquisition brings the original creators of Apache Iceberg and those of Linux Foundation Delta Lake, the two leading open source lakehouse formats, together. As one, they are going to lead the way with data compatibility so that you are no longer limited by which lakehouse format your data is in.</LI> <LI id="13c0" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph=""><STRONG class="lc fq">Lakeflow<SPAN> </SPAN></STRONG>helps you ingest data from databases, enterprise apps, and cloud sources, transform it into batch and real-time streaming, and confidently deploy and operate in production.</LI> <LI id="0441" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph=""><STRONG class="lc fq">Delta Lake 4.0</STRONG><SPAN> </SPAN>is the biggest release to date, with features for reliability, performance, and ease of use</LI> <LI id="6d8e" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph=""><STRONG class="lc fq">Apache Spark 4.0<SPAN> </SPAN></STRONG>There are a lot of exciting new features including ANSI mode by default, Python data source, polymorphic Python UDTF, string collation support, new VARIANT data type, streaming state store data source, structured logging, Java 17 by default, and many more</LI> </UL> <H1 id="dcfa" class="lz ma fp be mb mc md me mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw bj" data-selectable-paragraph="">Databricks + Tabular</H1> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nt nu ec nv bg nw" tabindex="0" role="button"> <DIV class="nk nl nm"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*S0I3rqdPatuzSkgZ.png 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*S0I3rqdPatuzSkgZ.png 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*S0I3rqdPatuzSkgZ.png 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*S0I3rqdPatuzSkgZ.png 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*S0I3rqdPatuzSkgZ.png 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*S0I3rqdPatuzSkgZ.png 1100w, https://miro.medium.com/v2/resize:fit:1400/format:webp/0*S0I3rqdPatuzSkgZ.png 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*S0I3rqdPatuzSkgZ.png 640w, https://miro.medium.com/v2/resize:fit:720/0*S0I3rqdPatuzSkgZ.png 720w, https://miro.medium.com/v2/resize:fit:750/0*S0I3rqdPatuzSkgZ.png 750w, https://miro.medium.com/v2/resize:fit:786/0*S0I3rqdPatuzSkgZ.png 786w, https://miro.medium.com/v2/resize:fit:828/0*S0I3rqdPatuzSkgZ.png 828w, https://miro.medium.com/v2/resize:fit:1100/0*S0I3rqdPatuzSkgZ.png 1100w, https://miro.medium.com/v2/resize:fit:1400/0*S0I3rqdPatuzSkgZ.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_0-1720712144510.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9500iB32B7E0FFEE2D8D9/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_0-1720712144510.png" alt="lara_rachidi_0-1720712144510.png" /></span> <P> </P> </DIV> </DIV> </FIGURE> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph="">This acquisition brings the original creators of Apache Iceberg and those of Linux Foundation Delta Lake, the two leading open source lakehouse formats, together.</P> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph="">The idea is to work closely with the Iceberg and Delta Lake communities to bring interoperability to the formats themselves. Ali (Databricks CEO) and Ryan (Tabular) shared their vision of formats being abstract from the user. If the formats become similar enough, it’ll be just an implementation detail that the engines will deal with.<BR />It’s a long journey, for sure, and that’s why Databricks introduced Delta Lake Uniform last year.<BR />Uniform tables provide interoperability across Iceberg, Delta Lake, and Hudi and support the Iceberg restful catalog interface so that companies can use the analytics engines and tools they’re already familiar with across all their data.</P> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph="">Both companies share a history of championing open source formats.<BR /><STRONG class="lc fq">TL;DR: Both formats will continue to coexist and should slowly converge over time so that it becomes an implementation detail and the user can focus on solving higher-value problems!</STRONG></P> <H1 id="0932" class="lz ma fp be mb mc md me mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw bj" data-selectable-paragraph="">Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering</H1> <P class="pw-post-body-paragraph la lb fp lc b ld mx lf lg lh my lj lk ll mz ln lo lp na lr ls lt nb lv lw lx fi bj" data-selectable-paragraph="">Lakeflow is a new solution that includes native and highly scalable connectors for the following tools:</P> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nk nl ny"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*kTle2mxIDJaONYb1.png 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*kTle2mxIDJaONYb1.png 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*kTle2mxIDJaONYb1.png 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*kTle2mxIDJaONYb1.png 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*kTle2mxIDJaONYb1.png 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*kTle2mxIDJaONYb1.png 1100w, https://miro.medium.com/v2/resize:fit:1076/format:webp/0*kTle2mxIDJaONYb1.png 1076w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 538px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*kTle2mxIDJaONYb1.png 640w, https://miro.medium.com/v2/resize:fit:720/0*kTle2mxIDJaONYb1.png 720w, https://miro.medium.com/v2/resize:fit:750/0*kTle2mxIDJaONYb1.png 750w, https://miro.medium.com/v2/resize:fit:786/0*kTle2mxIDJaONYb1.png 786w, https://miro.medium.com/v2/resize:fit:828/0*kTle2mxIDJaONYb1.png 828w, https://miro.medium.com/v2/resize:fit:1100/0*kTle2mxIDJaONYb1.png 1100w, https://miro.medium.com/v2/resize:fit:1076/0*kTle2mxIDJaONYb1.png 1076w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 538px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_1-1720712144294.jpeg" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9499i3DA9F1067C912A45/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_1-1720712144294.jpeg" alt="lara_rachidi_1-1720712144294.jpeg" /></span> <P> </P> </DIV> </FIGURE> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph="">Lakeflow is a unified solution for data ingestion, transformation and orchestration. Its three components are:</P> <UL class=""> <LI id="efbd" class="la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx nc nd ne bj" data-selectable-paragraph="">Lakeflow Connect: Simple and scalable data ingestion</LI> </UL> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nt nu ec nv bg nw" tabindex="0" role="button"> <DIV class="nk nl nz"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*UyEtf27SBnQYO64x.gif 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*UyEtf27SBnQYO64x.gif 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*UyEtf27SBnQYO64x.gif 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*UyEtf27SBnQYO64x.gif 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*UyEtf27SBnQYO64x.gif 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*UyEtf27SBnQYO64x.gif 1100w, https://miro.medium.com/v2/resize:fit:1400/format:webp/0*UyEtf27SBnQYO64x.gif 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*UyEtf27SBnQYO64x.gif 640w, https://miro.medium.com/v2/resize:fit:720/0*UyEtf27SBnQYO64x.gif 720w, https://miro.medium.com/v2/resize:fit:750/0*UyEtf27SBnQYO64x.gif 750w, https://miro.medium.com/v2/resize:fit:786/0*UyEtf27SBnQYO64x.gif 786w, https://miro.medium.com/v2/resize:fit:828/0*UyEtf27SBnQYO64x.gif 828w, https://miro.medium.com/v2/resize:fit:1100/0*UyEtf27SBnQYO64x.gif 1100w, https://miro.medium.com/v2/resize:fit:1400/0*UyEtf27SBnQYO64x.gif 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_2-1720712144642.gif" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9501i3091CE050D2CCA1D/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_2-1720712144642.gif" alt="lara_rachidi_2-1720712144642.gif" /></span> <P> </P> </DIV> </DIV> </FIGURE> <UL class=""> <LI id="1979" class="la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx nc nd ne bj" data-selectable-paragraph="">Lakeflow pipelines: Efficient declarative pipeline</LI> </UL> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nt nu ec nv bg nw" tabindex="0" role="button"> <DIV class="nk nl oa"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*YniQVoyzJy8Dy3ir.png 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*YniQVoyzJy8Dy3ir.png 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*YniQVoyzJy8Dy3ir.png 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*YniQVoyzJy8Dy3ir.png 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*YniQVoyzJy8Dy3ir.png 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*YniQVoyzJy8Dy3ir.png 1100w, https://miro.medium.com/v2/resize:fit:1400/format:webp/0*YniQVoyzJy8Dy3ir.png 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*YniQVoyzJy8Dy3ir.png 640w, https://miro.medium.com/v2/resize:fit:720/0*YniQVoyzJy8Dy3ir.png 720w, https://miro.medium.com/v2/resize:fit:750/0*YniQVoyzJy8Dy3ir.png 750w, https://miro.medium.com/v2/resize:fit:786/0*YniQVoyzJy8Dy3ir.png 786w, https://miro.medium.com/v2/resize:fit:828/0*YniQVoyzJy8Dy3ir.png 828w, https://miro.medium.com/v2/resize:fit:1100/0*YniQVoyzJy8Dy3ir.png 1100w, https://miro.medium.com/v2/resize:fit:1400/0*YniQVoyzJy8Dy3ir.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_3-1720712144960.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9502iDC69837DBB4A5A5D/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_3-1720712144960.png" alt="lara_rachidi_3-1720712144960.png" /></span> <P> </P> </DIV> </DIV> </FIGURE> <UL class=""> <LI id="a7b2" class="la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx nc nd ne bj" data-selectable-paragraph="">Lakeflow jobs: Reliable orchestration for every workload</LI> </UL> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph=""><A class="af ly" href="https://substack.com/redirect/9d44f215-4e50-43fa-8d83-418eff90d7aa?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Learn more on Databricks Lakeflow</A></P> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph=""><STRONG class="lc fq">Watch the announcement from Data+AI Summit 2024</STRONG></P> <FIGURE class="nn no np nq nr ns"> <DIV class="ob oc l ec"> <DIV class="od oe l"><IFRAME src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6rzQ6xjkYko%3Ffeature%3Doembed&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6rzQ6xjkYko&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6rzQ6xjkYko%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube" width="854" height="480" frameborder="0" scrolling="no" allowfullscreen="" class="el n fd dy bg" title="Databricks LakeFlow: A Unified, Intelligent Solution for Data Engineering. Presented by Bilal Aslam"></IFRAME></DIV> </DIV> </FIGURE> <H1 id="c358" class="lz ma fp be mb mc md me mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw bj" data-selectable-paragraph="">Delta Lake 4.0: The biggest release to date</H1> <P class="pw-post-body-paragraph la lb fp lc b ld mx lf lg lh my lj lk ll mz ln lo lp na lr ls lt nb lv lw lx fi bj" data-selectable-paragraph="">One of my favorite features is Delta Lake. This new version is the biggest release yet, with features for reliability and performance.</P> <UL class=""> <LI id="04c3" class="la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx nc nd ne bj" data-selectable-paragraph="">Delta Lake Uniform: allows Delta Lake tables to be read by Iceberg clients, enabling interoperability between the two formats without requiring data duplication or migration. UniForm automatically generates Iceberg metadata asynchronously, ensuring that Iceberg clients can read Delta tables seamlessly, while also supporting other formats like Hudi.</LI> <LI id="307f" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">When using Delta Uniform, you can take advantage of the most advanced features in Delta Lake like Liquid Clustering. Together those two features can provide amazing performance even when reading from Iceberg or Hudi.</LI> <LI id="bc70" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Variant Data Type (Parse_JSON): it’s a new feature designed to handle semi-structured data efficiently. It allows for flexible schema evolution without requiring a pre-defined schema, making it suitable for handling data with unknown or changing schema. The variant lib has been opensourced so that other formats and engines can share a common implementation!<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/09a79ffe-21bb-4bc9-be8e-3b135302d741?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Learn more</A></LI> <LI id="ac43" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Delta Connect: enables the use of Delta Lake with the decoupled client-server architecture of Spark Connect, allowing for more flexible and scalable data processing.</LI> <LI id="3580" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Safe Cross cloud writes: provides a central coordinator for commits, ensuring reliable and atomic writes across multiple clouds and engines.</LI> <LI id="cd32" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Type widening: is a feature that allows you to change the data type of a column in a Delta Lake table to a wider type without rewriting the underlying data files.<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/b6591126-e27f-4cd9-b48c-662369b48d5e?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Learn more</A></LI> </UL> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nt nu ec nv bg nw" tabindex="0" role="button"> <DIV class="nk nl of"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*bvRMLqjIlkC1NBeT.png 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*bvRMLqjIlkC1NBeT.png 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*bvRMLqjIlkC1NBeT.png 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*bvRMLqjIlkC1NBeT.png 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*bvRMLqjIlkC1NBeT.png 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*bvRMLqjIlkC1NBeT.png 1100w, https://miro.medium.com/v2/resize:fit:1400/format:webp/0*bvRMLqjIlkC1NBeT.png 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*bvRMLqjIlkC1NBeT.png 640w, https://miro.medium.com/v2/resize:fit:720/0*bvRMLqjIlkC1NBeT.png 720w, https://miro.medium.com/v2/resize:fit:750/0*bvRMLqjIlkC1NBeT.png 750w, https://miro.medium.com/v2/resize:fit:786/0*bvRMLqjIlkC1NBeT.png 786w, https://miro.medium.com/v2/resize:fit:828/0*bvRMLqjIlkC1NBeT.png 828w, https://miro.medium.com/v2/resize:fit:1100/0*bvRMLqjIlkC1NBeT.png 1100w, https://miro.medium.com/v2/resize:fit:1400/0*bvRMLqjIlkC1NBeT.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_4-1720712144762.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9504i6A736EA0A9E278B1/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_4-1720712144762.png" alt="lara_rachidi_4-1720712144762.png" /></span> <P> </P> </DIV> </DIV> </FIGURE> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph=""><STRONG class="lc fq">Watch the announcement from Data+AI Summit 2024</STRONG></P> <FIGURE class="nn no np nq nr ns"> <DIV class="ob oc l ec"> <DIV class="od oe l"><IFRAME src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fjoy4jdYJg3c%3Fstart%3D1%26feature%3Doembed%26start%3D1&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Djoy4jdYJg3c&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fjoy4jdYJg3c%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube" width="854" height="480" frameborder="0" scrolling="no" allowfullscreen="" class="el n fd dy bg" title="Announcing Delta Lake 4.0 with Liquid Clustering. Presented by Shant Hovsepian at Data + AI Summit"></IFRAME></DIV> </DIV> </FIGURE> <H1 id="f102" class="lz ma fp be mb mc md me mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw bj" data-selectable-paragraph="">Apache Spark 4.0: unified, ultra-Fast, scalable, flexible and powerful</H1> <UL class=""> <LI id="e7ee" class="la lb fp lc b ld mx lf lg lh my lj lk ll mz ln lo lp na lr ls lt nb lv lw lx nc nd ne bj" data-selectable-paragraph="">Spark Connect GA in Apache Spark 4.0 is a new client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. This decoupled architecture enables Spark and its open ecosystem to be leveraged from anywhere, embedded in modern data applications, IDEs, notebooks, and programming languages</LI> <LI id="3a31" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">ANSI mode is on by default in 4.0</LI> <LI id="1f1b" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Variant Data Type for semi-structure data</LI> <LI id="4629" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">String Collation Support</LI> <LI id="3def" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Millisec kafka 2 kafka latency for spark streaming!</LI> <LI id="5acc" class="la lb fp lc b ld nf lf lg lh ng lj lk ll nh ln lo lp ni lr ls lt nj lv lw lx nc nd ne bj" data-selectable-paragraph="">Streaming State data source which allows you to inspect the internal states of Streaming applications for debugging, profiling, testing, and troubleshooting</LI> </UL> <FIGURE class="nn no np nq nr ns nk nl paragraph-image"> <DIV class="nt nu ec nv bg nw" tabindex="0" role="button"> <DIV class="nk nl oa"><PICTURE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/format:webp/0*il79VJ51T_Bwy772.png 640w, https://miro.medium.com/v2/resize:fit:720/format:webp/0*il79VJ51T_Bwy772.png 720w, https://miro.medium.com/v2/resize:fit:750/format:webp/0*il79VJ51T_Bwy772.png 750w, https://miro.medium.com/v2/resize:fit:786/format:webp/0*il79VJ51T_Bwy772.png 786w, https://miro.medium.com/v2/resize:fit:828/format:webp/0*il79VJ51T_Bwy772.png 828w, https://miro.medium.com/v2/resize:fit:1100/format:webp/0*il79VJ51T_Bwy772.png 1100w, https://miro.medium.com/v2/resize:fit:1400/format:webp/0*il79VJ51T_Bwy772.png 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px"></SOURCE><SOURCE srcset="https://miro.medium.com/v2/resize:fit:640/0*il79VJ51T_Bwy772.png 640w, https://miro.medium.com/v2/resize:fit:720/0*il79VJ51T_Bwy772.png 720w, https://miro.medium.com/v2/resize:fit:750/0*il79VJ51T_Bwy772.png 750w, https://miro.medium.com/v2/resize:fit:786/0*il79VJ51T_Bwy772.png 786w, https://miro.medium.com/v2/resize:fit:828/0*il79VJ51T_Bwy772.png 828w, https://miro.medium.com/v2/resize:fit:1100/0*il79VJ51T_Bwy772.png 1100w, https://miro.medium.com/v2/resize:fit:1400/0*il79VJ51T_Bwy772.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og"></SOURCE></PICTURE><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="lara_rachidi_5-1720712144535.jpeg" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9503i9CC7BDE8960B6908/image-size/medium?v=v2&px=400" role="button" title="lara_rachidi_5-1720712144535.jpeg" alt="lara_rachidi_5-1720712144535.jpeg" /></span> <P> </P> </DIV> </DIV> </FIGURE> <FIGURE class="nn no np nq nr ns"> <DIV class="ob oc l ec"> <DIV class="od oe l"><IFRAME src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FS1B0J-uzSDE%3Fstart%3D1%26feature%3Doembed%26start%3D1&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DS1B0J-uzSDE&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FS1B0J-uzSDE%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube" width="854" height="480" frameborder="0" scrolling="no" allowfullscreen="" class="el n fd dy bg" title="What's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0"></IFRAME></DIV> </DIV> </FIGURE> <P class="pw-post-body-paragraph la lb fp lc b ld le lf lg lh li lj lk ll lm ln lo lp lq lr ls lt lu lv lw lx fi bj" data-selectable-paragraph="">Follow us on Linkedin:<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/8ccafbef-0c7a-4f92-8f4a-e681ef61e8cd?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Quentin</A><SPAN> </SPAN>&<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/ea0dac90-ae15-4c79-a46d-9fb4e4238148?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Youssef</A><SPAN> </SPAN>&<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/cbe1b0dd-a43e-4e46-bf59-e0366e328c6d?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Lara</A><SPAN> </SPAN>&<SPAN> </SPAN><A class="af ly" href="https://substack.com/redirect/5d853b27-3469-4b07-9c67-479ac62f662a?j=eyJ1IjoiMnoxdnlyIn0.fANyF2qulAoFrX8WJv7agMnp8-HXi1G4liF0rdRWYRE" target="_blank" rel="noopener ugc nofollow">Maria</A><SPAN> </SPAN>&<SPAN> </SPAN><A class="af ly" href="https://www.linkedin.com/in/beatrice-liew/" target="_blank" rel="noopener ugc nofollow">Beatrice</A></P> Tue, 23 Jul 2024 09:48:07 GMT lara_rachidi 2024-07-23T09:48:07Z Data+AI Summit 2024 Data Engineering Announcements https://community.databricks.com/t5/announcements/data-ai-summit-2024-data-engineering-announcements/m-p/78383#M220 <P><A class="af ly" href="https://nextgenlakehouse.substack.com/subscribe" target="_blank" rel="noopener ugc nofollow"><STRONG class="lc fq">Subscribe</STRONG></A><STRONG class="lc fq"><SPAN> </SPAN>to the NextGenLakehouse to receive monthly updates!</STRONG></P> Tue, 23 Jul 2024 09:48:07 GMT https://community.databricks.com/t5/announcements/data-ai-summit-2024-data-engineering-announcements/m-p/78383#M220 lara_rachidi 2024-07-23T09:48:07Z