Thanks Alicia! He may just be different... not as well "endowed" as the others. As long as he is functioning normally (not peeing on himself... soiling his belly, etc.) and there is no sign of infection, he should be fine.
If it's a very recent neuter, he could still just be sore.