Is there a way to programmatically clear the notebook state in Azure databricks?

- - Certifications
- - Learning Paths
- - Databricks Product Tours
- - Get Started Guides
- - Product Platform Updates
- - What's New in Databricks

- - Get Started Resources
- - Events
- - Support FAQs
- - Technical Blog
- - Knowledge Sharing Hub
- - Announcements
- - DatabricksTV

- - Private Groups
- - Skills@Scale

- - Databricks Community Champions
- - Khoros Community Forums Support (Not for Databricks Product Questions)
- - Databricks Community Code of Conduct

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I have a scenario where I have a series of jobs that are triggered in ADF, the jobs are not linked as such but the resulting temporally tables from each job takes up memory of the databricks cluster. If I can clear the notebook state, that would free up space for the next jobs to run. Any ideas how to programmatically do that woud be very mych apreciative.