Handling your rabbit is something you have to learn anyway. I like big breeds because they are more sturdy, and less likely to get under my feet. How difficult your rabbit is to handle doesn't just depend on size, but also on character. Sure, a small rabbit doesn't have the sheer power to slash you from shoulder to hip (=exaggeration, it's not that bad
, but I lost some T-shirts and blood) like mine have, on the other hand they are more fragile.
There's another breeder down the road who focus on pet rabbits - once she tossed 3 or 4 of her rabbits into my arms and they all were just "uhm, hi?", no issue at all. Only rabbits with such an attitude I ever met. My 10lbs farm rabbits on the other hand have a very strong opinion about being picked up, and it took me some years to get it right. Also met dwarfs that were virtually untouchable.
To pick a rabbit up you first have to get to it - if you force it on a rabbit, big or small, that will not work well. If you think that small rabbits are easier to pick up just because they are more easily overpowered, I don't think that will work, it will scare the rabbit, and break trust. Imho the size doesn't matter much in the end, there are bonuses and drawbacks to everything, you have to work with what you got.
I consider handling them one of the smaller topics concerning keeping rabbits. It's one of the things you'll have to learn anyway.