IE 11 is not supported. For an optimal experience visit our site on another browser.

Facebook helps the visually impaired 'hear' photos thanks to artificial intelligence

Starting today in a limited rollout, Facebook is using artificial intelligence to describe pictures to the visually impaired with a new feature.
/ Source: TODAY

Matt King has accomplished a lot in his life. He is a three-time Paralympian with a world record in tandem track cycling. He spent 25 years at IBM. And now, as an engineer — who happens to be blind — he is working to help Facebook users who are visually impaired “see” photographs on their phones.

Starting today in a limited rollout, the Menlo Park, California-based company is using artificial intelligence to describe pictures to the visually impaired with a new feature called automatic alternative text. King, 50, told TODAY that it’s “a major step towards equal access to information.”

Using a database of millions of images, Facebook’s “visual recognition engine” picks out people and objects in photos, and can even provide context as to where an image was taken.

RELATED: Twitter CEO Jack Dorsey on whether platform censors users: 'Absolutely not'

Imagine, for example, a picture taken at a beach. Facebook can tell you how many people are in the photo, whether they are smiling, other objects (such as a boat or sunglasses) that might be in the frame, and that it was taken outdoors. It can even identify whether a photo is a selfie or a meme.

“Facebook's mission is to make the world more open and connected and that goes for everyone, including the visually, hearing or physically impaired,” the company said in a statement.

Now a member of Facebook’s accessibility engineering team, King said that the automatic alternative text project appealed to him from the moment he heard about it.

Matt King, center right, is a Facebook engineer who is helping to make the social network more accessible to people who are visually impaired.Matt Toder/

“It was in the works at Facebook before I arrived and I was immediately interested in it, because of its importance to the blind community,” King told TODAY. “I also felt strongly about contributing the perspective of a blind person to design of the product.”

RELATED: How to land an internship at Facebook: Be persistent, crave knowledge

So far, Facebook told TODAY, feedback from blind users has been “really positive.” There are 39 million people in the world who are blind, and roughly 285 million people who are visually impaired, according to the World Health Organization. Now, they will be able to experience the joys of actively engaging with countless photos of their friends’ pets and babies.

But Facebook doesn’t want to rush this technology or over-promise its capabilities. The feature will only identify objects if it’s 80 percent certain that it’s correct. Even then, each statement is prefaced with the words “image may contain” to clarify that Facebook isn’t always right. Over time the company will updates its database of images to increase accuracy and create more detailed descriptions of photos.

It initially launches for users of the iPhone’s English-language screen reader in the United States, U.K., Canada, Australia and New Zealand, but will be available in other platforms, languages and countries “soon,” according to Facebook.

RELATED: How to make your phone battery last longer in 5 easy steps

In a related move last week, Twitter now allows users to add alt text to images in Tweets.

Facebook’s software shouldn’t be a surprise to anyone who has watched CEO Mark Zuckerberg declare his intense interest in artificial intelligence. Facebook has teams of employees in California, New York City and Paris committed to the technology, and Zuckerberg even expressed his desire for a digital assistant like Jarvis from “Iron Man.” The same technology that powers Facebook’s automatic alternative text is also used to improve search results and take down offensive content.

Engineer Matt King, left, and Jeff Wieland, who heads up Facebook's accessibility teamMatt Toder/

Other companies, such as Google, Microsoft and IBM, are researching artificial intelligence as well. The goal isn’t to create super-smart machines that will take over the world, as movies like “Terminator” might lead people to believe. Instead, it’s to process massive amounts of data to give researchers and users information that they can use.

For blind users of Facebook, where more than 350 million photos are uploaded every day, this technology could be a very big deal. Even those with perfect eyesight will benefit from the fact that “the photos they're posting can be understood by a broader audience,” King said. “We care about including and connecting everyone.”