With feeding unlimited pellets and hay, have you measured out the amount that she is actually consuming in a day and compared it with her weight? A non pregnant/non nursing 5 lb rabbit should be eating approximately 1/4-1/2 cup pellets and a pile of hay the size of their body, in a day. Generally about an ounce of dry food per pound of body weight. Though this will change now that she has kits, as the amount she will need to eat will at least triple, depending on how many kits she has. So she will be consuming quite a bit more than 1/2 cup pellets per 5 lb body weight as she starts nursing and the babies start growing. The amount she needs to be eating will gradually increase.
Also check the amount of water she is drinking in a day. If she drinks from a bottle, I would offer her a water dish as well, as rabbits tend to drink better from them, and the more water they are able to drink will affect the amount of food they eat as well.
If she isn't eating an adequate amount for her size, then she could have something like dental problems preventing her from eating enough. If she is eating enough for her body weight and starting to nurse, she could have something like parasites, or there are serious health issues that can cause weight loss; like hepatic coccidiosis, kidney and liver disease, and cancer. So if she is consuming plenty of food for her size and starting to nurse, then I would suggest taking her to an experienced rabbit vet to get a fecal float done to check for parasites, her teeth checked to rule out dental issues, and if those don't show anything being wrong you may need blood work and xrays done.
http://www.rabbitnetwork.org/educat...abbit-health-articles/weight-loss-in-rabbits/
https://rabbit.org/vet-listings/
In the meantime, getting her switched onto the alfalfa based pellets will help, not only with the increased protein, but also she will need the higher calcium content for nursing. I would also consider maybe adding in a bit of alfalfa hay to help with weight gain and calcium levels. It's important to ensure a nursing doe has adequate dietary calcium intake to prevent low blood calcium from occurring(very dangerous). I've also used black oil sunflower seeds to help with weight gain for my old rabbit that was losing weight.