Yeah, nails are always a problem. The only rabbit I've had who found a solution is Aki: she hates being picked up and she hates me cutting her nails more, so after a few years she began to cut her own nails - seriously, she just cut them with her teeth and I haven't had to trim her nails for about 3 years now. I took a look at them just last week and they were short and perfect. Unfortunately, Aki is an isolated case as she's unusually clean and bright (I swear she's more clever than my dog and she has crazy good intuitions about things).
I always do Tybalt's nails sitting on the floor with his bum resting on my knees so it's not too scary. I try not to carry my rabbits at my own height when I can help it and to stay as close to the floor as possible (to make the experience easier and to avoid injuries in case the rabbit succeed in escaping - it also prevents me from holding too tight as it's no big deal if the rabbit escapes, I'm really paranoid when it comes to accidentally crushing something by pressing too hard on their bellies). There is a position, slightly reclined back, when he stops moving and it's easier to do him (I use it a lot these days as I have to put cream under his paws twice a day for his slight pododermatitis) but it's not simple to find it the first times especially since you have to be very slow and careful not to hurt your bunny's back.
Some people put the rabbit in a towel and make a burrito. I could never figure out how you successfully do something like that, but I'm not very good at manual things.
Still, there are some rabbits with whom it's hard to find any good solution. I've had a rabbit who was a nightmare when it came to nails and would completely panic everytime, but he was scared of heights so just being picked up triggered wild movements and the occasional scream.