article How to migrate from mount points to Unity Catalog Volumes in Technical Blog

article How to migrate from mount points to Unity Catalog Volumes in Technical Blog https://community.databricks.com/t5/technical-blog/how-to-migrate-from-mount-points-to-unity-catalog-volumes/ba-p/64422 <P><SPAN>Mount points are a convenient pattern to facilitate data access in Databricks, while hiding the complexity of accessing cloud storage directly. Unity Catalog improves the governance of file-based cloud storage with Volumes. </SPAN></P> <P><SPAN>In this blog post, we use an example to discuss the steps involved in planning and migrating to Volumes.<BR /><BR /></SPAN></P> <H1><SPAN>Mount points link remote storage to local storage</SPAN></H1> <P><SPAN>The Databricks </SPAN><A href="https://docs.databricks.com/en/dbfs/mounts.html" target="_blank" rel="noopener"><SPAN>documentation</SPAN></A><SPAN> explains mount points as enablers for users “to mount cloud object storage to the Databricks File System (DBFS) to simplify data access patterns for users that are unfamiliar with cloud concepts.” Now, imagine you need to access unstructured data from cloud storage managed by another team or another external party. How would access control work in practice?</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_0-1713540376458.png" style="width: 465px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7162iC70F9B482986CD49/image-dimensions/465x416?v=v2" width="465" height="416" role="button" title="lammic_0-1713540376458.png" alt="lammic_0-1713540376458.png" /></span></P> <P><SPAN>Let’s take AWS and S3 storage as an example: Databricks clusters leverage IAM roles to access the different mount points. Therefore, any user on the Databricks workspace with access to the respective clusters can now access these mount points. In Azure, you create mount points via Microsoft Entra ID application service principals; once mounted, all users will have access to that storage account. When rotating the service principal secret, you must unmount and remount the storage.</SPAN></P> <P>Unity Catalog on the Databricks Intelligence Platform offers a higher, finer-grained level of governance compared to mount points. With mount points, access control is managed through compute configurations, typically overseen by central IT teams. This approach ties access to the compute resources rather than the data itself, affecting governance. By migrating to Unity Catalog Volumes, you can grant direct access to the data assets, enabling more decentralized governance.</P> <P> </P> <H1><SPAN>Unity Catalog governs volumes</SPAN></H1> <P><SPAN>To close this gap, Unity Catalog introduces the concept of </SPAN><A href="https://www.databricks.com/blog/announcing-general-availability-unity-catalog-volumes" target="_blank" rel="noopener"><SPAN>Volumes</SPAN></A><SPAN>. Volumes extend the governance layer in the lakehouse from tabular data to </SPAN><I><SPAN>any</SPAN></I><SPAN> data on storage. This is a critical capability when, for example, building AI applications that source various data shapes and formats.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_1-1713540376489.png" style="width: 705px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7164i8772C221EA8A3A7B/image-dimensions/705x369?v=v2" width="705" height="369" role="button" title="lammic_1-1713540376489.png" alt="lammic_1-1713540376489.png" /></span></P> <P> </P> <H1><SPAN>Migration process</SPAN></H1> <P><SPAN>Business fit and continuity need to be at the heart of every migration strategy, and good planning is therefore crucial. In times before Unity Catalog, mount points were often used for a twofold purpose:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>To read tabular data from paths to the object storage. In Unity Catalog, this pattern is covered by </SPAN><A href="https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html" target="_blank" rel="noopener"><SPAN>external tables</SPAN></A><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>To read unstructured data in the object storage. In Unity Catalog, this pattern is covered by the concept of <A href="https://www.databricks.com/blog/announcing-general-availability-unity-catalog-volumes" target="_blank" rel="noopener">Volumes</A>.</SPAN></LI> </UL> <P><SPAN>In the following sections, we will explore in more detail how to <EM>assess, plan, and execute</EM> our migration from mounts to Unity Catalog Volumes.<BR /><BR /></SPAN></P> <TABLE style="border-color: #FFF8DC; width: 100%; background-color: #fff8dc; border-style: hidden;" border="1" width="100%"> <TBODY> <TR> <TD width="100%" height="166px"><SPAN><STRONG><BR />Tip!<BR /></STRONG><BR />Databricks Labs has released </SPAN><A href="https://github.com/databrickslabs/ucx" target="_blank" rel="noopener"><SPAN>UCX</SPAN></A><SPAN>, a companion tool for upgrading to Unity Catalog. UCX automates and simplifies migrations to Unity Catalog; UCX does not cover volumes yet (see this </SPAN><A href="https://github.com/databrickslabs/ucx/issues/716" target="_blank" rel="noopener"><SPAN>feature request</SPAN></A><SPAN>). Until that feature is available, please look at the steps outlined in the “execution” section below.<BR /><BR /></SPAN></TD> </TR> </TBODY> </TABLE> <H2><SPAN><BR />Assessment</SPAN></H2> <P><SPAN>Before you migrate, you need a high-level overview of </SPAN><I><SPAN>what</SPAN></I><SPAN> to migrate. The assessment phase consists of collating information about the existing mount points including but not limited to: the underlying storage location, impacted code assets (e.g. notebooks, queries, etc.), and where each mount point is being used. </SPAN></P> <P><SPAN>You can get information about mount points via </SPAN><FONT face="courier new,courier"><SPAN>dbutils.fs.mounts()</SPAN></FONT><SPAN> or by running the UCX assessment. Listing the assets requires “grepping” (searching) across the code base.</SPAN></P> <P><SPAN>Example output below:<BR /><BR /></SPAN></P> <TABLE style="width: 250px;" width="250"> <TBODY> <TR> <TD width="16.44325290437891%" height="50px"> <P><STRONG>Mount point</STRONG></P> </TD> <TD width="39.946380697050934%" height="50px"> <P><STRONG>Cloud storage location</STRONG></P> </TD> <TD width="43.52100089365505%" height="50px"> <P><STRONG>Code assets impacted (notebooks, etc.)</STRONG></P> </TD> </TR> <TR> <TD width="16.44325290437891%" height="80px"> <P><FONT face="courier new,courier">/mnt/domainAimages</FONT></P> </TD> <TD width="39.946380697050934%" height="80px"> <P><FONT face="courier new,courier"><SPAN>s3://one-blog-post-mount-location-one/images/</SPAN></FONT></P> </TD> <TD width="43.52100089365505%" height="80px"> <P><FONT face="courier new,courier"><SPAN>/Workspace/Repos/michele@databricks.com/image-analysis</SPAN></FONT></P> </TD> </TR> <TR> <TD width="16.44325290437891%" height="80px"> <P><FONT face="courier new,courier"><SPAN>/mnt/domainBdata</SPAN></FONT></P> </TD> <TD width="39.946380697050934%" height="80px"> <P><FONT face="courier new,courier"><SPAN>abfss://domainb@oneblogpost.dfs.core.windows.net/</SPAN></FONT></P> </TD> <TD width="43.52100089365505%" height="80px"> <P><FONT face="courier new,courier"><SPAN>/Workspace/Repos/michele@databricks.com/read-raw-blobs</SPAN></FONT></P> </TD> </TR> <TR> <TD width="16.44325290437891%" height="51px"> <P><FONT face="courier new,courier"><SPAN>…</SPAN></FONT></P> </TD> <TD width="39.946380697050934%" height="51px"> <P><FONT face="courier new,courier"><SPAN>…</SPAN></FONT></P> </TD> <TD width="43.52100089365505%" height="51px"> <P><SPAN>…</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P> </P> <H2><SPAN>Planning</SPAN></H2> <P><SPAN>For each mount point, you will need to decide:</SPAN></P> <UL> <LI><SPAN>Whether you still need it! There’s no need to migrate mount points that are not in use anymore</SPAN></LI> <LI><SPAN>Whether the capability is better served and governed as an external table or volume in the Unity Catalog</SPAN></LI> </UL> <P><SPAN>The discriminant is the content, whether it is structured or unstructured data. For unstructured data, mark it as Volume (V); for structured data, mark it as External Table (ET).</SPAN></P> <P><SPAN>Depending on the number of mount points, storage locations, and business considerations, you may execute the migration in one step or multiple phases. For each mount point, indicate which phase you plan to migrate it in.</SPAN></P> <P><EM>Example:<BR /><BR /></EM></P> <TABLE style="width: 100%;"> <TBODY> <TR> <TD width="183.828px" height="50px"> <P><STRONG>Mount point</STRONG></P> </TD> <TD width="188.531px" height="50px"> <P><STRONG>Target (ET / V / remove)</STRONG></P> </TD> <TD width="58.1406px" height="50px"> <P><STRONG>Phase</STRONG></P> </TD> </TR> <TR> <TD width="183.828px" height="51px"> <P><FONT face="courier new,courier"><SPAN>/mnt/domainAimages</SPAN></FONT></P> </TD> <TD width="188.531px" height="51px"> <P><FONT face="courier new,courier"><SPAN>V</SPAN></FONT></P> </TD> <TD width="58.1406px" height="51px"> <P><FONT face="courier new,courier"><SPAN>1</SPAN></FONT></P> </TD> </TR> <TR> <TD width="183.828px" height="51px"> <P><FONT face="courier new,courier"><SPAN>/mnt/domainBdata</SPAN></FONT></P> </TD> <TD width="188.531px" height="51px"> <P><FONT face="courier new,courier"><SPAN>ET</SPAN></FONT></P> </TD> <TD width="58.1406px" height="51px"> <P><FONT face="courier new,courier"><SPAN>2</SPAN></FONT></P> </TD> </TR> <TR> <TD width="183.828px" height="51px"> <P><FONT face="courier new,courier"><SPAN>…</SPAN></FONT></P> </TD> <TD width="188.531px" height="51px"> <P><FONT face="courier new,courier"><SPAN>…</SPAN></FONT></P> </TD> <TD width="58.1406px" height="51px"> <P><FONT face="courier new,courier"><SPAN>…</SPAN></FONT></P> </TD> </TR> </TBODY> </TABLE> <P> </P> <H2><SPAN>Execution</SPAN></H2> <P>To help illustrate the steps involved in migrating from mounts to UC Volumes, let us paint the below scenario. </P> <P class="lia-indent-padding-left-30px"><EM>You have already built an LLM chatbot with RAG that leverages internal confidential documents stored in PDF format. These documents are available from an external location and are currently accessed as a DBFS mount point. </EM></P> <P>By migrating to UC Volumes, you will gain finer grained access control on these confidential documents and are able to (via <A href="https://docs.databricks.com/en/data-governance/unity-catalog/data-lineage.html" target="_blank" rel="noopener"><SPAN>data lineage)</SPAN></A> audit the flow of information to keep track of how the PDF content feeds the LLM chatbot.</P> <P><SPAN>In the current configuration, the PDF documents are available under the mount point</SPAN></P> <P class="lia-indent-padding-left-30px"><FONT face="courier new,courier"><SPAN>/mnt/michele/</SPAN></FONT></P> <P><SPAN>which is pointing to the Amazon S3 location</SPAN></P> <P class="lia-indent-padding-left-30px"><FONT face="courier new,courier"><SPAN>s3://one-blog-post-mount-location/michele/<BR /><BR /></SPAN></FONT></P> <TABLE style="border-color: #FFF8DC; width: 100%; background-color: #fff8dc; border-style: hidden;" border="1" width="100%"> <TBODY> <TR> <TD width="100%" height="166px"><SPAN><STRONG>Tip!<BR /></STRONG><BR /></SPAN><SPAN>Use a table to keep track of mount points and their <I>new</I> Volume equivalent. Keeping track of progress helps you programmatically test and update code sections later.<BR /></SPAN></TD> </TR> </TBODY> </TABLE> <P><SPAN>You should also treat this as an opportunity to simplify access. </SPAN><SPAN>For example, if multiple mount points share the same root (at the storage level); you can re-use the same storage credential for multiple Volumes.<BR /><BR /></SPAN></P> <H3><SPAN>1. Set up storage credentials</SPAN></H3> <P><SPAN>To migrate toward a UC Volumes, you need to first make sure you can access the data from the external cloud storage location. To do this, you create a <A href="https://docs.databricks.com/en/sql/language-manual/sql-ref-storage-credentials.html" target="_self">storage credential</A> as in the example below: </SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_2-1713540376490.png" style="width: 574px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7163i66E577498D3108ED/image-dimensions/574x676?v=v2" width="574" height="676" role="button" title="lammic_2-1713540376490.png" alt="lammic_2-1713540376490.png" /></span></P> <P><SPAN>Here, you provide an authentication and authorization mechanism which is dependent on the cloud provider. On AWS you would use an AWS IAM role, on Azure a managed identity, or on Google Cloud a service account.<BR /><BR /></SPAN></P> <H3><SPAN>2. Set up external locations</SPAN></H3> <P><SPAN>Next, you need to create the relevant external locations, as in the example below: </SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_3-1713540376392.png" style="width: 578px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7165i0EE60FD3A08F712F/image-dimensions/578x616?v=v2" width="578" height="616" role="button" title="lammic_3-1713540376392.png" alt="lammic_3-1713540376392.png" /></span></P> <P><SPAN>Then, verify you can access the external location.<BR /></SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_4-1713540376492.png" style="width: 589px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7166i8351E61B30070E4B/image-dimensions/589x324?v=v2" width="589" height="324" role="button" title="lammic_4-1713540376492.png" alt="lammic_4-1713540376492.png" /></span></P> <H3><SPAN>3. Create UC Volume</SPAN></H3> <P><SPAN>Once connection has been established successfully, you can create the UC volume with:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_5-1713540376493.png" style="width: 572px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7167iC03BA1D51474AEA7/image-dimensions/572x580?v=v2" width="572" height="580" role="button" title="lammic_5-1713540376493.png" alt="lammic_5-1713540376493.png" /></span></P> <P><SPAN>And verify its creation:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_6-1713540376467.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7168i36B25F1AA1C3FDB3/image-size/medium?v=v2&px=400" role="button" title="lammic_6-1713540376467.png" alt="lammic_6-1713540376467.png" /></span></P> <P><SPAN>Notice that the UC volume location is a subfolder of the external location. You can reuse the same external location to create multiple volumes as subfolders. You can manage permissions for each volume separately, even though they share the same parent location.<BR /><BR /></SPAN></P> <TABLE style="border-color: #d6eeee; width: 100%; background-color: #d6eeee; border-style: hidden;" border="1" width="100%"> <TBODY> <TR> <TD width="100%" height="166px"><SPAN><STRONG>Note!<BR /><BR /></STRONG>The UC volume location is a subfolder of the external location. You can reuse the same external location to create multiple volumes as subfolders. You can then manage permissions for each volume separately, even though they share the same parent location.</SPAN><SPAN><BR /></SPAN></TD> </TR> </TBODY> </TABLE> <P><SPAN>At this stage, you can access the same cloud storage location in S3 via mount point </SPAN><I><SPAN>and</SPAN></I><SPAN> UC (external volume). If you upload a file via UC volume, the file will be available via the mount point and vice versa. This allows you to migrate the mount points with zero downtime for your application.<BR /><BR /></SPAN></P> <H3><SPAN>4. Adjust code and clean up</SPAN></H3> <P><SPAN>With your volume in place, you now need to update your code to use the volume instead of the mount point. If there are only a few code artifacts, it might be simpler and quicker to do this manually.</SPAN></P> <P><EM>Example:</EM></P> <P><SPAN>Reading the PDF file as Spark DataFrame with mount points:</SPAN></P> <P> </P> <LI-CODE lang="python">pdf_df = spark.read.format('binaryFile').load('/mnt/michele/TenThings_Supplemental.pdf')</LI-CODE> <P> </P> <P><SPAN>After the migration to UC volume:</SPAN></P> <P> </P> <LI-CODE lang="python">pdf_df = spark.read.format('binaryFile').load('/Volumes/michele/default/externalvolume/TenThings_Supplemental.pdf')</LI-CODE> <P> </P> <P><SPAN>However, if you have multiple notebooks and files in your workspace, please consider scanning all of them and replacing the respective paths until UCX implements the following </SPAN><A href="https://github.com/databrickslabs/ucx/issues/391" target="_blank" rel="noopener"><SPAN>feature request</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>Use the mapping table you created during the planning phase to track your migration progress as you eliminate the need for each mount point, clean up permissions, and build storage credentials. Depending on your cloud provider, you will also need to ensure the IAM roles or Managed Identities have the <EM>least required privileges</EM> to storage. </SPAN></P> <P><SPAN>It’s always good practice to use parameters in your code; it’s even more critical for UC volume paths.<BR /><BR /></SPAN></P> <TABLE style="border-color: #d6eeee; width: 100%; background-color: #d6eeee; border-style: hidden;" border="1" width="100%"> <TBODY> <TR> <TD width="100%" height="166px"><SPAN><STRONG><BR />Important!<BR /></STRONG><BR />Mount points have workspace-level scope, while UC volumes have metastore-level scope. For example, if your development, test, and production workspaces are bound to the same Unity Catalog metastore, their respective volumes must have different names. <BR /><STRONG><BR /></STRONG></SPAN></TD> </TR> </TBODY> </TABLE> <H1><SPAN><BR />Conclusion</SPAN></H1> <P><SPAN>We just walked through the basic steps for migrating from mount points to volumes in Unity Catalog. As opposed to the passthrough nature of mount points, volumes - like external tables - ensure fine-grained access control and provide enhanced data governance capabilities including lineage.</SPAN></P> <P><SPAN>You can also decentralize access control by moving it from a central team (in charge of infrastructure management) to the business teams (the actual data owners). By removing friction, you can reduce the time to market for your projects.</SPAN></P> <P><SPAN>As you plan your migration, put your platform users’ experience at the center of everything and ensure business continuity. Then, automate as much as possible when migrating and leveraging the UCX framework.</SPAN></P> Thu, 02 May 2024 12:58:34 GMT lammic 2024-05-02T12:58:34Z How to migrate from mount points to Unity Catalog Volumes https://community.databricks.com/t5/technical-blog/how-to-migrate-from-mount-points-to-unity-catalog-volumes/ba-p/64422 <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="lammic_1-1713540376489.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7164i8772C221EA8A3A7B/image-size/large?v=v2&px=999" role="button" title="lammic_1-1713540376489.png" alt="lammic_1-1713540376489.png" /></span></P> Thu, 02 May 2024 12:58:34 GMT https://community.databricks.com/t5/technical-blog/how-to-migrate-from-mount-points-to-unity-catalog-volumes/ba-p/64422 lammic 2024-05-02T12:58:34Z