topic Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used? in Data Engineering

Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

wpenfold — Thu, 10 Feb 2022 21:45:32 GMT

Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

Anonymous — Thu, 10 Feb 2022 21:49:25 GMT

You can see when a notebook was last run if it's attached to an active cluster. You can also read old logs to see what happened, but it's a lot of work for almost no gain. There isn't any harm in having old notebooks that aren't run. I have some notebooks in a workspace I have never run once and it's not problematic

Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

wpenfold — Thu, 10 Feb 2022 21:54:05 GMT

Hi Josephk, I'm new to databricks, but I've been asked to clean up old notebooks in our environment that have been created over the years, and are no longer used. Is there an API I can use to find when the last time a notebook was run? Or any other suggestion you have?

Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

Anonymous — Thu, 10 Feb 2022 23:56:49 GMT

I looked around internally and couldn't find anything. Certainly nothing in the docs. Maybe just try deleting things and seeing if people complain?

Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

AmanSehgal — Fri, 11 Feb 2022 00:39:16 GMT

Using workspace API you can list out all the notebooks for a given user.

The API response will tell you if the objects under the path is a folder or a notebook.

If it's a folder then you can add it to the path and get notebooks within the folder.

Put all of that in an excel or something and ask your team members if they need a notebook or not.

GET https://<databricks-host-name>/api/2.0/workspace/list
 
Body:
 
{ "path": "/Users/<username>" }

Refer to this documentation for more details.

Also, for a period of 'x' months archive them all in a github repo, in case someone needs access to notebooks later.

Going ahead, add sufficient logs in the notebook or a mechanism to record execution time.

It could be as simple as an insert statement at top cell that inserts a row in a table default.notebook-run with values notebook-name and timestamp, every time a notebook runs.

Re: Is there a way I can tell when a Notebook was last run, so I can identify and delete Notebooks that are no longer being used?

wpenfold — Wed, 16 Feb 2022 18:48:33 GMT

Just wondering...I can display 'Recent Activity' for a notebook--which gives me the information I'm looking for. So it is being collected...someplace. I can't find it in the APIs. Anyplace else I could look for that info?