If a vet won't spay, only neuters, I would highly suspect them of not being very experienced with rabbits.
The main reasons to neuter a male is because of unwanted hormonal behaviors or plans on bonding with another rabbit.. Reasons for females is hormonal behavior, plans on bonding, and the risk of reproductive cancer, which older females are at higher risk of developing. For male rabbits, while they certainly can develop testicular cancer, there are no studies(that I've seen) to suggest they are at high risk of developing it. The ones I've seen say it's uncommon, except for males with undescended testicles. Google 'testicular cancer in rabbits study' and it will bring up science articles on the subject.
As long as he is an only rabbit and has no unwanted hormonal behaviors, doesn't have an undescended testicle, and the low cancer risk isn't a concern to you, then it's just your personal choice whether you want to have it done or not.