cluster sharing between different notebooks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2024 04:50 PM
I have two structured streaming notebooks running continuously for anomaly detection. Both notebooks import the same python module to mount the Azure blob storage, but each has its own container. Each notebook runs well when it has its own cluster. When I run both notebooks on the same cluster, error happened. Debugging shows that I am using the same variable name in two notebooks for ADLS storage path:
adls_path='abfss://%s@%s.dfs.core.windows.net/' % (container_name, account_name)
Since the container name is different for each notebook, the adls_path will be different too.
When I run both notebooks on the same cluster, how does databricks set adls_path? Will there be a race condition? Is there any way to fix the issue without giving different variable name?
