As someone who handles dead mice on an almost daily basis (yay snake food!), the worst you really have to worry about with mice is salmonella. Treat it like you do raw chicken and you'll be fine as far as germs go.

That said, mouse eviscera smells disproportionally horrid. Every so often one I'm warming up (I buy them frozen) will get opened, and after disposing of it, I have to leave the room for an hour or so with the windows open to air it out. So over-bleaching might not be medically necessary, but it's sure as hell good for keeping the apartment habitable.