It's not frivolous stuff. Looking after your own mental and physical health also directly benefits the household.

It is one thing to say "Oh I should finish cleaning up after dinner before I go relax for an hour" versus "The kitchen hasn't been cleaned in a while so I should sacrifice the time I set aside for myself to deal with that."

I'm not saying you need to viciously defend all of your unscheduled time, but I do think you need to actively make time to do things for yourself that are hobbies or are relaxing. It won't hurt the household - if it is that dependent on you, there is already a problem.