10-29-2021 07:10 AM
I have a databricks job on E2 architecture in which I want to retrieve the workspace instance name within a notebook running in a Job cluster context so that I can use it further in my use case.
While the call
dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply("browserHostName")
works in an All-Purpose cluster in an interactive way, this key does not exist when running in a Job cluster context.
10-29-2021 07:43 AM
it is not possible to get the workspace URL/Shard name from MT/E2 because we don't inject the user-visible URL, but the driver reachable one.
But as a workaround to get the shardname in our job, we can write a file with our shard name in a particular dbfs location and read it inside the job as shown below:
dbutils.fs.put("/FileStore/Deepak-test/ShardName.txt", "https://cse2.cloud.databricks.com/")
When you will run a job, you can read that file and get the shard name from there.
dbutils.fs.head("/FileStore/Deepak-test/ShardName.txt")
Or you can create a dictionary as well which you can use to get shardname inside a job
WORKSPACES = {
"111": "workspace1",
"222": "workspace1"
}
org_id = get_notebook_context().tags().apply("orgId")
print(f"https://{WORKSPACES[org_id]}.cloud.databricks.com")
04-28-2022 01:02 AM
Actually is anybody aware of a fix for Azure Databricks, too? (My use case is to send out status messages from a batch run which includes a link to the job cluster page or run page to allow operators to click for details.)
05-25-2022 12:26 AM
Found workaround for Azure Databricks question above: dbutils.notebook.getContext().apiUrl will return the regional URI, but this forwards to the workspace-specific one if the workspace id is specified with o=.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.