topic Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster in Data Engineering

Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster

Deepak_Bhutada — Fri, 29 Oct 2021 14:10:40 GMT

I have a databricks job on E2 architecture in which I want to retrieve the workspace instance name within a notebook running in a Job cluster context so that I can use it further in my use case.

While the call

dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply("browserHostName")

works in an All-Purpose cluster in an interactive way, this key does not exist when running in a Job cluster context.

Re: Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster

Deepak_Bhutada — Fri, 29 Oct 2021 14:43:54 GMT

it is not possible to get the workspace URL/Shard name from MT/E2 because we don't inject the user-visible URL, but the driver reachable one.

But as a workaround to get the shardname in our job, we can write a file with our shard name in a particular dbfs location and read it inside the job as shown below:

dbutils.fs.put("/FileStore/Deepak-test/ShardName.txt", "https://cse2.cloud.databricks.com/")

When you will run a job, you can read that file and get the shard name from there.

dbutils.fs.head("/FileStore/Deepak-test/ShardName.txt")

Or you can create a dictionary as well which you can use to get shardname inside a job

WORKSPACES = {
"111": "workspace1",
"222": "workspace1"
}
 
org_id = get_notebook_context().tags().apply("orgId")
 
print(f"https://{WORKSPACES[org_id]}.cloud.databricks.com")

Re: Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster

Thomas_B_ — Thu, 28 Apr 2022 08:02:50 GMT

Actually is anybody aware of a fix for Azure Databricks, too? (My use case is to send out status messages from a batch run which includes a link to the job cluster page or run page to allow operators to click for details.)

Re: Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster

Thomas_B_ — Wed, 25 May 2022 07:26:21 GMT

Found workaround for Azure Databricks question above: dbutils.notebook.getContext().apiUrl will return the regional URI, but this forwards to the workspace-specific one if the workspace id is specified with o=.