Here's a link to the NFFGRB article page, lots of good info on there, a number of the articles mention color and its relation to size. The articles on Blues and Light Grays are really informative. Some of the articles bring up using Sandies, Whites and Fawns to increase size, bone and quality, folks can read these and come to their own conclusions.
http://www.nffgrb.com/FlemishArticles.html
I've actually seen a 25lb White that was sitting on a scale, it belonged to John Long. It was one huge bunny, not fat, just massive, great deal of bone. His Whites in general were quite large. Very important point brought up by Blue Giants is fertility in the larger rabbits, while my light gray does were very fertile and producing large litters into their third year my friends much larger Sandies did not. Seems the larger they get, the sooner fertility drops off. Old time breeders that I know told me that most does fertility in the big ones dropped off at 2, so thats when they sold off their senior does.
I do pick my rabbits based on color first, I think a lot of people do, and its a big deal in Flemish because there's such a stigma on crossing many of the colors. If your selling a Light Gray it better not have Sandies on the ped, you'll lose a sale, and get a reputation as someone who crosses the wrong colors. Same thing in Silvers, in fact the three colors are all very distinctly different from one another, in some ways almost like three different breeds. Fawn Silvers are smaller, fine boned, elfish almost, Blacks are bigger, better fleshed and solid, big difference between the two. In Mini Rex Castors, REW and Blacks are most often picked for Best of Breed, and say, BEW are still a work in progress and not up to the quality of some of the other colors. But the difference in MR is the lack of a stigma in crossing colors, so its easier and quicker to improve the newer color varieties. I know that there are color vs size/quality differences in many breeds, not just Flemish, and these seem to be more marked in breeds that have strong color crossing rules. Of course its just based on averages, your always going to have above and below average individuals.
Very interesting point, there's someone working on Opal Flemish, thats going to really freak some folks out. I hope it will eventually be accepted as a new color though.