Yah, this is true and I know that but would be nice all in the same spot - really my idea came from wanting to leave a listener in place but not delete it, only disable - and then I noticed the edit vh interface is designed the same way.
Maybe you can just put a marker in the config file (db?) instead of deleting it entirely.
Obviously not a high priority request but something to consider the next time the UI is being tweaked.