I have a deaf bunny. My suspicions started in just the same way. Harold is VERY curious and interested in what is going on so it surprised me that he didn't respond to me calling him or clapping or any noises. So I tried to test him a little by clapping,turning on the vacuum right behind him... His ears don't even twitch.
My other rabbit, however, is laid back, but does respond to noises like clapping and other bangs by at least turning her ears towards the sound. When I took Harold to the vet for another problem I asked her about the deafness and she said if I was suspicious of deafness then he was probably was deaf. Rabbits are generally highly responsive to noise so your suspicions are probably correct.
Harold's deafness hasn't caused us any problems. He's very friendly and affectionate and doesn't spook easily, so we haven't had problems scaring him when he doesn't know we're coming. He's learned some hand signals to come and to go into his cage. If we can't find him usually stomping on the floor will bring him out. I think he can feel the vibrations, because clapping, calling, etc. doesn't work as it does with Judy. We hardly even notice that he's different anymore.
Good luck with Mr. Fluffy! Try some more tests and watch for ear twitching. Perhaps he is just a very chill bunny.
Another great testis rattling the food bag or some other food related noise... If he ignores that then he probably can't hear ya.