Get the latest from TODAY

Sign up for our newsletter
By Keith Wagstaff

You can't judge a book by its cover. It turns out humans are not very good at following that advice — a behavior that affects who we hire, who we elect to office and even who we send to jail.

"We are, at best, very weak at being able to guess things about people from their faces," Christopher Olivola, who researches the psychology of human decision making at Carnegie Mellon University, told TODAY. "The problem is, once we see a person's face, we put so much weight on that cue that we end up ignoring other, more useful pieces of information."

In a recent study in the journal "Trends in Cognitive Science," Olivola and two Princeton University researchers looked at how people judge others based on facial features. What they found was strong evidence of "face-ism." Essentially, we decide that someone is trustworthy or competent or extroverted based solely on their facial features.

Facial expressions demonstrating competence (A) and dominance (B), ranging left to right from lower than average, average, and higher than average.Today

People who look more mature are inherently judged as competent; baby-faced folks are not. 

And "Zoolander" was right. Being really, really, ridiculously good-looking helped across every category. 

"Attractiveness, generally speaking, is always a plus," Olivola said. 

The study specifically looked at people's resting faces. That means a scowl or a smile provoked by emotion didn't factor into it — the researchers only looked at faces in "neutral" positions. 

Like many people, Olivola has a problem: he naturally kind of looks upset or angry. Or, as it is sometimes called, bitchy resting face.

"When I’m neutral, I look really mean — thanks mom and dad!" he said. "But people just find me intimidating. Like when I was growing up, people would come up to me and say, 'Hey is everything OK? And I would say, 'No I feel fine.'"

This is a real thing; some people were consistently rated by test subjects as either angry or happy. The naturally angry-looking people were rated less trustworthy, while the happy-looking people were given the benefit of the doubt. Olivola and his team got good enough at determining how people would judge a face that they could take someone's picture and adjust the level of competence or trustworthiness with a slider. 

All of this superficial behavior is perfectly natural. Apes make similar judgments, Olivola said. 

"We have been doing things like processing faces far longer than we have been reading or processing numbers."

But we aren't just using faces to judge people on the street. 

His studies have shown that people will vote for candidates who look more competent or decide to convict people who look less trustworthy. Changing our natural instincts is tough, but we could do things like make sure voters are more informed — which makes them less likely to vote based on appearances, Olivola said — and hide defendants in trials from the view of jurors. 

"This facial stereotyping, or face-ism, affects really big societal choices, like who we vote for, who we choose to handle our money and who we convict in a crime," he said. "We don’t want to live in a society where these things are determined by superficial cues."

Keith Wagstaff writes about technology for NBC News. He previously covered technology for TIME's Techland and wrote about politics as a staff writer at TheWeek.com. You can follow him on Twitter at @kwagstaff and reach him by email at: Keith.Wagstaff@nbcuni.com