I think it depends on how long you will be gone for. If you are gone for up to 4 days, then I would not choose to bring him unless you cannot find a pet sitter. If you will be gone longer than 4 days, then it would be easier on him to take him. Some rabbit do take change better than others. If he stresses after a short trip (like to the vet), then a long trip is probably not the best. If he is Ok with shorter trips and want to explore new places, then he would do better with a longer trip.
I have taken my buns on a 3.5 hour trip (one way) for a day trip for bunny hopping. We ended up having to do it all in one day, so they spent quite a bit of time in the carriers. My rabbits are fairly used to travelling, but this was a lot in one day, they are more used to 30-45 min car rides one way and a few hours at the place before going home. We do this at least once a month, some months are more. At the event where we go, I do have a pen for the rabbits to hang out it and they get to do agility as excercise during the day. They also have food and water at the event and a bit in the carrier.
Assuming that you end up having to bring him along, you should get him used to the carrier and begin in the car. I would first get him used to the carrier, then just being in the car, then a short trip, then a longer trip. If there is a safe place near you, you can see how he does in a new environment after being in the carrier. You can even try giving him hay or treats in the carrier to help him get used to it. Put a soft blanket in the carrier and have food and water dishes in it too. I put hay and sometimes veggies in. Hay is for eating and the veggies help with hydration as water bottles and bowls and leak.
For the actual trip, you should bring an pen or safe place for him to be while you are there. Also bring food and water from home as different places have different water and you might not be able to get his usual food there. Have dishes that won't break during travel, but are easy to keep clean, if he uses a water bottle, have 2 just in case. Bring some blankets that have his scent on them so he has something that is the same. Bring a litter box, litter, some bags and other litter cleaning supplies, bring enough litter for the trip plus some extra. Toys are also good to bring.
It is a good idea to get the info of any rabbit savvy vets in the area just in case. You don't want to be scrambling for that info in an emergency. A small first aid kit is also helpful.