Do it! I gives him some extra time with you, gets him used to being in the car, around other people, scents, etc. it's really good to get your rabbits out, if not too stressful, so that they are more socialized.
This will help as they get older so they are not fearful of people, places, etc...