I'm don't an expert on this procedure whatsoever, but with a quick Google search I found 4-6 months is when it's safe to get him neutered. Of course, if you have a experienced rabbit vet he/she will be able to give you the best age to fix your rabbit.
I feel your pain though, I have a father...