Physical Rest vs. Mental Rest
Thinking about work is still work. It can still drain you mentally so that when you return to work from physical rest, you're still just as tired.
It doesn't feel like you had enough of a break because you didn't.
Take your mind on vacation too. Let it just exist in the moment instead of worrying.
You'll get the most out of your break time that way.