Rabbit won't let me pick him up anymore

[caudilad]

I adopted Oz about a month ago, when I first got him he would let me pick him up but now when I try to pick him up he growls at me. I want to get a harness for him but I'm afraid if he won't let me pick him up he won't let me put it on him. He's already been neutered so I don't think it's hormones any ideas why he won't let me pick him up anymore