I have been told by loads of people not to handle them yet so.....
I don't know who is telling you this, but many of the people that have commented on this thread about handling the kits, are experienced rabbit breeders that know what they are talking about. Checking the babies daily is an important part of raising baby rabbits. You want to be checking the babies daily to ensure they are eating and no injuries or health problems have occurred. These daily checks also ensure they are used to being handled by people, and don't grow up nervous or scared of it.
You do want to be careful when handling them as they are very squirmy and can pop right out of your hand, so you want to handle them close to a surface or floor so that they can't get injured from a possible fall. It's also a good idea to wash your hands before handling. You can then pet the mom to get her scent on your hands before picking them up if you want. It may help if she is a bit nervous about her babies being handled.