It is not available natively in Databricks. But you can write an administration script that analyzes your jobs data and automatically cleans up the older jobs as needed. It would be easiest to do this with the jobs API.
- List your jobs to get all the job ids
- For each job id you would could either get the latest job runs to see if there are any jobs that don't have any runs OR you would use the created datetime in the job data to delete old data.
Please note that job runs are only stored for 60 days.
