I think this is the limitation and now we have no way to do this via the UI or CLI

The way to do it depends on the type of backend file store that you are using.

If you are using file store it is easy as the deleted experiment is stored in the trash folder

.t
 
rm -rf mlruns/.trash/*