Databricks Engineering New Features - May 2024

lara_rachidi — Fri, 14 Jun 2024 17:24:50 GMT

- Liquid Clustering is GA: Liquid Clustering replaces table partitioning and ZORDER, simplifying data layout and optimizing query performance without needing fine-tuning. It allows flexible redefinition of clustering keys without data rewrites, evolving with analytic needs. Blog Post: https://www.databricks.com/blog/announcing-general-availability-liquid-clustering?utm_source=bambu&utm_medium=social&utm_campaign=advocacy - Ingesting XML Data in Public Preview: Native support for XML files is now available in Auto Loader and COPY INTO, enabling direct ingestion, querying, and parsing without external packages. This includes features like schema inference, evolution, and XML-specific capabilities like row-level validation against XSD and the from_xml Spark function. Blog Post: https://www.databricks.com/blog/announcing-simplified-xml-data-ingestion#:~:text=(Note%20that%20Auto%20Loader%20for,and%20evolution%20in%20Auto%20Loader. - Bind storage credentials and external locations to specific workspaces (Public Preview): You can now bind storage credentials and external locations to specific workspaces, preventing access to those objects from other workspaces. This feature is especially helpful if you use workspaces to isolate user data access, for example, if you have separate production and development workspaces or a dedicated workspace for handling sensitive data. Documentation: https://docs.databricks.com/en/connect/unity-catalog/external-locations.html#workspace-binding - Databricks Git folders are GA: a visual Git client and API, supports common Git operations, enabling code development in notebooks with version control, collaboration, and CI/CD. Documentation: https://docs.databricks.com/en/repos/index.html - Private Exchange: This is a new feature on the Databricks Marketplace that simplifies the process of setting up private exchanges for all Databricks customers. For example, if you’re a multinational conglomerate with multiple subsidiaries, you do not want to license or monetize your data with the broader world, but you’d like to share data securely with a private network of external partners, such as partner retailers, suppliers, and distributors. Link to blog: https://www.databricks.com/blog/unlocking-potential-private-data-sharing-using-databricks-private-exchanges?utm_source=bambu&utm_medium=social&utm_campaign=advocacy

article Databricks Engineering New Features - May 2024 in Databricks TV

Databricks Engineering New Features - May 2024