What you decide to do is your choice. Nobody else's. No one on here knows you, knows your life, knows your struggles and difficulties, so they can't truly see what is going to be the best thing for you. This is a forum and everyone on here is just stating an opinion. And that's all it is, an opinion. Not what you should do, not what's the right thing for you. Set the guilt aside, you have nothing to feel guilty about, and there's nothing you should be beating yourself up over. This is the normal process in finding a potential companion for a single bun. Except for the not being able to pick the potential bunny companion. Normally you should have a say as to which bun you want to bring home.
The fact that you weren't able to actually choose which bun seemed the best fit for you, as well as being the right fit for your girl bun, makes it a more difficult situation, and one there is no reason for you to feel bad about. You got him to be able to be a companion for your bun, and if that bond isn't working then the situation can't and shouldn't be forced. That's part of the reason to go to a shelter or rescue in the first place when trying to find a companion for your rabbit. So that you will be able to make the right match.
If he's not feeling like the right match, see what other buns they have that could be. If you aren't sure yet, then no harm in giving it a few more weeks. But one thing, I would suggest is trying a different approach for you. Don't worry about trying to make a connection with him, that's not so important right now. Just do the essential care, and keep your energy focused on finding out whether or not he can be a good companion for your girl. But it can take some patience at the start. Keep in mind that a bit of chasing, humping, and nipping(but not actual fighting) can all be a normal part of the bonding process and is not at all a reflection of what their final bond can end up being. It's a normal part of rabbits sorting out hierarchy. Once that gets all sorted out, you could have two buns that end up totally adoring each other. And if that happens, I think then you could learn to love him too.
But if the bonding process isn't going well and they don't seem the right fit, then it's a kindness to return him so that your bun can find a companion she is happy with, and that he can also go to a home and possibly be bonded to a bun he will adore and be happy with as well.