We rescued a female rabbit - abandoned on a country road - that may have come from a mink or fur farm. I took her in for her spay procedure. For the longest time, she would bond and allow hubby to hold her and pick her up, and Regina readily jumped in hub's lap.
A suspicion is that a human female in her past could have conjured up unpleasant and fearful memories wrt handling or if the human female came 'round to remove rabbits from cages.
There is nothing wrong with you.
I saw this at the shelter. Rabbits can detect human hormonal scents, etc.
Our elder gal _gradually_ learned to trust me. Yet for years she would dig, nip, and not allow me to cuddle or hold her on my lap. I got nipped. Hubby? No, Regina preferred hubby's affection.