Could facial recognition technology have caught the Boston bombers?

While ubiquitous surveillance cameras certainly played a key role in the apprehension of Boston Marathon bombers Dzhokhar and Tamerlan Tsarnaev, the way investigators finally identified the brothers was pretty old-fashioned: they had a victim identify the suspects from a series of photos, released the images to the public, and waited for the tips to come pouring in. As a number of articles pointed out in the aftermath, this is not how it works on TV, where cops are always yelling "enhance" at big video screens and running surveillance images against a massive database to identify the bad guys. 

Despite what you've seen on CSI, we don't quite have the technology to do that yet, but it may not be far off. Michigan State computer scientists Joshua Klontz and Anil Jain recently put out a study testing whether existing facial recognition software could have ID'd the Tsarnaevs based on the security camera images taken just before the bombing.

They added three headshots of each of the Tsarnaevs brothers to a database of over a million mugshots from the Pinellas County, Florida Sherrif's Office. The images of the Tsarnaevs included Tamerlan's booking photo from a 2009 domestic violence arrest. The program then returned ranked matches from the database for the surveillance images. The rankings below are from the program NeoFace 3.1: 


As you can see, the program got one bull's eye based on Dzhokhar's high school graduation photo. Unfortunately for law enforcement purposes, Tamerlan's mugshot, an image that would presumably have been available on a police database, came back ranked extremely low -- in 116,342nd place as a match the first surveillance photo and in 471,165th for the second. (PittPatt, another commercial recognition software which was acquired by Google in 2011, did not perform as well.)

In general, the software performed better on images of Dzokhar than Tamerlan, who was wearing sunglasses in the footage from the bombing site: thus the supposed resemblance to the mulleted gentleman in the top row. When the researchers conducted a filtered search -- looking only at white men in their 20s -- the software did better -- returning four matches in the top 30 for Dzhokhar, though the graduation photo is still the only top match:

So, could the bombing investigators have used this software to identify the Tsarnaevs? Well, no. Dzhokhar's high school graduation photo wouldn't have been in a government database. (Although this image, which was released by the FBI prior to Dzhokhar's arrest and looks like it may have been a driver's license photo, was ranked 19th on one of the filtered searches so perhaps it could have helped narrow the search down using other information.) The programs also seem to have a problem with sunglasses and eyepatches. 

But all the same, this is pretty new technology and that no. 1 hit on Dzhokar does suggest we may be getting closer to this kind of thing actually being useful. 

Photo provided by FBI via Getty Images