Ephemeral storage how to create/mount.

[cszczotka]

Hi,

I'm looking for information how to create/mount ephemeral storage to Databricks driver node in Azure Cloud.  

Does anyone have any experience working with ephemeral storage?

Thanks,

[Kaniz]

Hi @cszczotka, 

• Azure Databricks allows you to mount cloud object storage to the Databricks File System (DBFS) to simplify data access patterns for users who are unfamiliar with cloud concepts.

• Mounted data does not work with Unity Catalog, and Databricks recommends migrating away from using m...¹.

• To mount cloud object storage, you create a link between your workspace and the storage. This enables you to interact with the storage using familiar file paths relative to the DBFS.

• The following Python code snippet demonstrates how to mount storage:

  dbutils.fs.mount(
      source="your_storage_uri",
      mount_point="/mnt/your_mount_point",
      encryption_type="",  # Optional encryption type
      extra_configs=None  # Optional additional configuration specifications
  )
  

• Remember to check with your workspace and cloud administrators before configuring or altering data m...

• The ephemeral storage attached to the driver node is block storage with native POSIX-based path access.
• Any data stored in this location disappears when a cluster terminates or restarts
• When working with ephemeral storage, consider the following best practices:
  • Use Unity Catalog volumes to configure access to non-tabular data files stored in cloud object storage.
  • For ML frameworks and other OSS Python modules that require FUSE, use POSIX-style paths via Unity Catalog volumes or DBFS mounts.
  • Avoid modifying a mount point while other jobs are reading or writing to it. After modifying a mount, run dbutils.fs.refreshMounts() on all other running clusters to propagate a....
• If you have any further questions, feel free to ask! 😊