topic Databricks Governance and DWH Announcements — July 2024 in Announcements

topic Databricks Governance and DWH Announcements — July 2024 in Announcements https://community.databricks.com/t5/announcements/databricks-governance-and-dwh-announcements-july-2024/m-p/83550#M224 <P><STRONG><FONT size="5">Governance & Delta Sharing</FONT></STRONG></P> <P><STRONG>New connection support for Lakehouse federation</STRONG></P> <P>You can run federated queries on data managed by Salesforce Data Cloud</P> <DIV class="captioned-image-container"> <FIGURE> <DIV class="image2-inset"><SPAN>You can use SSO authentication to connect to SQL Server</SPAN></DIV> </FIGURE> </DIV> <P><FONT size="4"><STRONG>Query History and Node timeline are now available in System Tables</STRONG></FONT></P> <UL> <LI> <P><SPAN>Query history show sql queries run by SQL warehouses. It’s powerful to track your queries (execution status, error, data management etc). </SPAN><A href="https://docs.databricks.com/en/admin/system-tables/query-history.html" rel="" target="_blank">Learn more</A></P> </LI> <LI> <P><SPAN>Node timeline captures node-level resource utilization data at minute granularity. Each record contains data for a given minute of time per instance. </SPAN><A href="https://docs.databricks.com/en/admin/system-tables/compute.html#node-timeline" rel="" target="_blank">Learn more</A></P> </LI> </UL> <DIV class="pencraft pc-display-flex pc-alignItems-center pc-position-absolute pc-reset header-anchor-parent"> </DIV> <P><STRONG>Partition metadata logging for UC external tables</STRONG></P> <DIV class="pencraft pc-display-flex pc-alignItems-center pc-position-absolute pc-reset header-anchor-parent"> <DIV class="pencraft pc-display-contents pc-reset _pubTheme_ipix0_1"><SPAN>For DBR13.3 LTS or above, you can enable partition metadata logging, which is a partition discovery strategy for external tables registered to Unity Catalog. The behavior is consistent with the partition discovery strategy used in Hive metastore and only impacts Unity Catalog external tables that have partitions and use Parquet, ORC, CSV, or JSON.</SPAN></DIV> </DIV> <P>Databricks recommends enabling the new behavior for improved read speeds and query performance for these tables<STRONG>.<A href="https://docs.databricks.com/en/tables/external-partition-discovery.html" rel="" target="_blank">Learn more</A></STRONG></P> <P><STRONG>What’s new in Delta Sharing ?</STRONG></P> <P>Delta Sharing lets you share tables where liquid clustering is enabled as well as the objects metadata including comments and primary constraints and AI Models.</P> <P><STRONG><A href="https://docs.databricks.com/en/delta/clustering.html" rel="" target="_blank">Liquid Clustering Sharing Support</A> <A href="https://docs.databricks.com/en/data-sharing/create-share.html#models" rel="" target="_blank">AI Models Sharing</A></STRONG></P> <P><FONT size="5"><STRONG>Compute & Data Engineering</STRONG></FONT></P> <P><STRONG>Databricks Lakeflow Connect is available</STRONG></P> <P>LakeFlow Connect offers native connectors that enable you to ingest data from databases and enterprise applications and load it into Databricks. LakeFlow Connect leverages efficient incremental reads and writes to make data ingestion faster, scalable, and more cost-efficient, while your data remains fresh for downstream consumption.</P> <P>Salesforce Sales Cloud, Microsoft Azure SQL Database, Amazon RDS for SQL Server, and Workday Reports-as-a-Service (RaaS) are currently supported.</P> <P><STRONG><A href="https://www.databricks.com/blog/ingest-data-sql-server-salesforce-and-workday-lakeflow-connect" rel="" target="_blank">Blog Announcement</A>, <A href="https://www.youtube.com/watch?v=6rzQ6xjkYko" rel="" target="_blank">Keynote presentation</A></STRONG></P> <P><STRONG>Databricks Serverless Compute for Workflows and Notebooks are GA</STRONG></P> <P>Configuring and managing compute such as Spark clusters has long been a challenge for data engineers and data scientists. Time spent on configuring and managing compute is time not spent providing value to the business.</P> <P><STRONG>Databricks Connect for Python now supports Serverless Compute</STRONG></P> <P><LI-WRAPPER></LI-WRAPPER></P> <H4 class="header-anchor-post"> </H4> <DIV id="youtube2-D7aXvBCt3C0" class="youtube-wrap" data-attrs="{"videoId":"D7aXvBCt3C0","startTime":null,"endTime":null}" data-component-name="Youtube2ToDOM"> <DIV class="youtube-inner"><STRONG><IFRAME src="https://www.youtube-nocookie.com/embed/D7aXvBCt3C0?rel=0&autoplay=0&showinfo=0&enablejsapi=0" width="728" height="409" frameborder="0" allowfullscreen="true" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true"></IFRAME></STRONG></DIV> <DIV class="youtube-inner"> <P><FONT size="5"><STRONG>Data Warehousing</STRONG></FONT></P> <P><SPAN>Predictive Optimisation, which can improve your query performance by 2x through intelligent optimization of data layouts, is now in GA. </SPAN><A href="https://www.databricks.com/blog/announcing-general-availability-predictive-optimization" rel="" target="_blank">Learn more</A></P> <P><SPAN>Cost management dashboards are now in Public preview, making it easy to import a dashboard to monitor costs on a workspace or account level. </SPAN><A href="https://docs.databricks.com/en/admin/account-settings/usage.html#import-a-usage-dashboard" rel="" target="_blank">Learn more</A></P> <P> </P> <FONT size="5"><STRONG>In a nutshell</STRONG></FONT> <UL> <LI> <P>In Databricks Runtime 15.4 LTS and above, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs)</P> </LI> <LI> <P><SPAN>Account SCIM v2.1. </SPAN><A href="https://docs.databricks.com/en/_extras/documents/account-scim-2_1.pdf" rel="" target="_blank">Learn more</A></P> </LI> <LI> <P><SPAN>End of life for Databricks managed passwords. </SPAN><A href="https://docs.databricks.com/en/security/auth/password-deprecation.html" rel="" target="_blank">Learn more</A></P> </LI> <LI> <P><SPAN>Databricks provides an open source software (OSS) JDBC driver that enables you to connect tools such as </SPAN><A href="https://docs.databricks.com/en/dev-tools/datagrip.html" rel="" target="_blank">DataGrip</A><SPAN>, </SPAN><A href="https://docs.databricks.com/en/dev-tools/dbeaver.html" rel="" target="_blank">DBeaver</A><SPAN>, and </SPAN><A href="https://docs.databricks.com/en/partners/bi/workbenchj.html" rel="" target="_blank">SQL Workbench/J</A><SPAN> to Databricks through Java Database Connectivity (JDBC), an industry-standard specification for accessing database management systems.</SPAN></P> </LI> <LI> <P>Library installation on clusters now has a timeout of 2 hours.</P> </LI> </UL> </DIV> </DIV> Tue, 20 Aug 2024 07:03:02 GMT youssefmrini 2024-08-20T07:03:02Z Databricks Governance and DWH Announcements — July 2024 https://community.databricks.com/t5/announcements/databricks-governance-and-dwh-announcements-july-2024/m-p/83550#M224 <UL> <LI> <P><STRONG>Serverless Compute</STRONG><SPAN> for Workflows and Notebooks are GA</SPAN></P> </LI> <LI> <P><STRONG>Databricks Lakeflow Connect is now available</STRONG></P> </LI> </UL> Tue, 20 Aug 2024 07:03:02 GMT https://community.databricks.com/t5/announcements/databricks-governance-and-dwh-announcements-july-2024/m-p/83550#M224 youssefmrini 2024-08-20T07:03:02Z