[JimMorrison]

I'm pretty sure that there is no fancy targeting algorithm.

Fire Closest will target the closest, of course. Any other targeting selection will select an entirely random squad of the specified type. It tends to be harder to see than with melee orders, but they both function in the same way.