IMO Vengeance of the Dead is buggy too, for including undead kills in its dead count. It's the souls of the dead you slew coming back to haunt you. So when you double up, where the heck did those extra undead come from?
I think it's unfair to cast this again if the unit you are trying to kill survives. Once is fair game though.
Vengeance does have an MR check, so you should do what you can to increase your MR to reduce your chances of being affected.
Undead are immune, so if you're really desperate you could use Twiceborn and kill yourself.