Get the latest from TODAY

Sign up for our newsletter
By Kelley Holland

Summer means vacation for the vast majority of Americans. But more and more often, those days off aren't really downtime.

The job market may be improving, but many people may still feel edgy about completely uncoupling themselves from the office. Purestock / Today

Some 61 percent of Americans plan to work during vacation this year, according to a new survey by Harris Interactive. That's up from the 52 percent found in a similar survey in 2012 and 46 the year before that.

That's a lot of downtime converted to working hours. A recent survey of affluent investors by John Hancock found that 80 percent plan to take a summer vacation.

(Read more: Most Americans plan to travel for summer vacation)

So are these working vacationers happy warriors? Not exactly. A third of the survey respondents who plan to work are not pleased about it – and 4 percent intend to deal with their frustration by throwing something. (Six percent plan to use their vacation to update their resume.)

One problem is that people aren't just working while they're on vacation: they're working on weekends, too. The latest American Time Use Survey from the Bureau of Labor Statistics found that a third of the highest paid workers – those earning $1,291 a week or more – worked on an average Saturday, Sunday or holiday, for an average of 3.6 hours.

(Read more: No paid vacation? You must be an American)

The job market may be improving, but many people may still feel edgy about completely uncoupling themselves from the office.

(Read more: Job growth posts large gain)

People are also staying plugged in because they are afraid of being overwhelmed when they return to work, according to Ellen Galinsky, president of the Families and Work Institute. "It's less the big bad people making us do it. It's more ourselves and worrying what will happen when we don't," she said.

And the Harris Interactive survey suggests things aren't likely to change anytime soon. The propensity to work during vacation is greatest for Generation Y: 73 percent of respondents aged 18 to 34 said they expect to work during their time off this year.

While staying plugged in may ease your first day back, it's not really a good thing, Galinsky said. People need downtime and a change of scene to really recharge.

"Why do you get your best ideas in the shower, or when you are walking the dog?" Galinsky asked. Stepping back from work "gives you a chance to pull together things that as we're rushing through we don't necessarily see together."

People become more productive, she said, "having a time where you don't feel like you're on a treadmill all the time – or at least you're on a treadmill of your own choosing at a spa."

More from TODAY Money:

Follow business on Twitter and Facebook