What happened to the ephemeral notebook links????? and the job ids????

[Benedetta]

Hey Databricks,

      Why did you remove the ephemeral notebook links and job Ids from the parallel runs? This has created a huge gap for us. We can no longer view the ephemeral notebooks, and also the Jobids are missing from the output. Waccha doing? The link that is provided in the output is the link to the notebook that was called.....not the run of the notebook, aka the ephemeral notebook. This is no longer troubleshootable. Please restore the jobids and ephemeral notebooks.

Thank you,

Benedetta

[Kaniz_Fatma]

Hi @Benedetta, I understand your frustration, and it’s important to have clear visibility into job runs for troubleshooting purposes.

Let me provide some information to help address this issue.

In Databricks, ephemeral notebooks are temporary instances of a notebook that are created when you run a job. These notebooks allow you to view the exact code and output produced during the job execution. Similarly, job IDs uniquely identify each job run, making it easier to track and troubleshoot any issues.

Here are some steps you can take to work with ephemeral notebooks and job IDs:

1. Job URLs and IDs:

   • To get the job URL and ID, navigate to the Workflows section in the Databricks sidebar. Click on the specific job name, and you’ll find the job ID in the URL after the text #job/. The job URL is essential for troubleshooting failed job runs¹.
   • For example, in the following screenshot, the job ID is 1: ²(https://westus.azuredatabricks.net/?o=6280049833385130)

2. Notebook Parameters:

   • If you’re running notebooks via jobs, consider using notebook parameters. You can pass parameters to your notebook runs, which can help you differentiate between different job executions.
   • In the latest Databricks CLI, you can provide notebook parameters using the --notebook-params flag. For example:

     databricks jobs run-now --job-id '256794483973904' --notebook-params '{"key1": "value1"}'
     

   • Alternatively, create a JSON file with the required parameters and provide the parameter file path u...³.

3. Job Run ID within Notebooks:

   • If your notebook has been triggered by dbutils.notebook.run, you can find the job ID within the notebook context. Look for the tag "jobId" in the context JSON.
   • You can then return the job ID using dbutils.notebook.exit(job_id)⁴.

Remember, technology evolves, and sometimes changes can be frustrating. Hang in there, and I hope you find a solution that works well for your use case! 😊🚀

[Benedetta]

Hi Kaniz,

    It's funny you mention these things - we are doing some of those - the problem now is that the JobId is obscured from the output meaning we can't tell which ephemeral notebook goes with which JobId.  It looks like the ephemeral notebook is available via the "StartTime" or "EndTime" columns which I had removed from the output due to the fact that I could not view the parameters. If I remove all of the columns, except parameters, I can then view the parameters and using CTRL-F, potentially match the outputs. Without removing the columns, the parameters really can't be seen. I don't know why it was necessary to obscure the JobIds, or even how the parameters are really useful given how difficult they are to see, with respect to many notebook runs in parallel. The JobIds are readily available - why not bring them back??

Thank you,

Benedetta