Bug? Rule I am unaware of? Am I just stupid?

[Yvelina]

I was besieging a castle with a few units. I 'Astral traveled' a few troops and commanders in the province. The defenders sallied and my forces routed. It should have not been a big deal, as no one important was killed. Instead I had all of the teleported troops die with a 'no where to retreat' message. I lost a prophet, quite a few artifacts, etc... The strange thing is that the troops that had swam into the province, or that had been there before the attack retreated just fine.

Is this a bug (some troops dying, others retreating) or am I somehow not understanding what happened?