Happiness is mandatory

In the world of the role-playing game Paranoia, The Computer oversees all citizens of a futuristic city called Alpha Complex, including a squad of "Troubleshooters," whose job is to shoot any troublemakers (like Communists and "The Outdoors") trying to disrupt its algorithmically perfect society. In a reboot of the game due out later in 2019 (or, "when Friend Computer feels like it"), there is another directive at play to keep the peace of Alpha Complex: happiness is mandatory.

Many dystopian fictional scenarios have incorporated mass surveillance and strict mandates on emotional expression, from Big Brother in Orwell's 1984, to the emotionally blunted population of 2002's Equilibrium. In an emotional surveillance state, not only are they always watching, they are also always telling you to smile.

But what if I told you we're already living in this type of dystopia – that your facial recognition data can be harnessed by any networked device with a camera, that your face most likely exists in a database already that allows your emotions to be analyzed instantly, and even if you covered up your webcam and never looked directly at your phone, you still may come face-to-face (so to speak) with a billboard that changes based on whether you smile or sneer at it in the near future?

The technology is already here, and the emotional surveillance state is closer than you may think.


Face the music

In 2015, the market for facial recognition technology was valued at $2.77 billion, and as the industry has grown, estimates for 2020 have ballooned from $6.19 to $22 billion. This inflation of the estimation is likely due to two major factors: how much better the technology has become in the meantime, and how many companies have entered the facial data processing game, ensuring both more data and more advances in what can be done with it, as well as a healthy competition between the world's largest tech companies.

Unless you've escaped the reach of Facebook, Apple, Microsoft, Google, Amazon, Disney, or Snapchat in at least the last 5 years, it's time to face the music: they've already got your face.

Almost every major player in the tech scene has some skin in:

In 2014, Facebook launched DeepFace, which was able to determine whether two pictures were of the same person with 97.25% accuracy (while humans scored 97.53%). Anyone who has used the "suggested tag" feature knows it may not always seem that accurate, but with more voluntarily-given pictures of faces in their book than any company should have any right to, Facebook has always been in the perfect position to turn your profile picture into profit.


Apple wasn't far behind in the face arms race, though, filing a patent in 2014 for AI technology that could analyze and identify mood based on facial expressions. In early 2016, they acquired Emotient, a facial analysis and emotion recognition software company, and while TIME Magazine guessed at the time that they would use the technology to improve image recognition in the photo library, it has clearly also made its way into the iPhone X's Animoji and Face ID.

Microsoft debuted their facial recognition software by Microsoft Project Oxford in 2015. Their Microsoft Face API has been used since then in Microsoft Hello, their equivalent to Apple's Face ID, and they even offer a free demo of their emotion recognition technology online:

Screenshot of Microsoft AI Face and Emotion Recognition demo (Face portraits courtesy of thispersondoesnotexist.com)

Google also debuted their facial recognition technology called FaceNet in 2015, scoring 100% accuracy on a test to label faces in the wild, and has continued to add facial detection, recognition, and emotional content analysis to their Cloud Vision API services.

Amazon's computer vision division has Rekognition, which not only scans for facial recognition and emotion, but could be used to recognize 100 people in a single image and match them to a database of tens of millions.

Disney also has access to at least 16 million faces of moviegoers, thanks to a collaboration with Caltech in processing emotional data in real time, and Snapchat filed a patent for emotion recognition AI in early 2018, likely using the treasure trove of face data gleaned from over 300 million active users per month trying on rainbow vomit and dog ear filters.

So unless you've escaped the reach of Facebook, Apple, Microsoft, Google, Amazon, Disney, or Snapchat in at least the last 5 years, it's time to face the music: they've already got your face.

And these are just the companies that you've probably heard of in the US, UK, or EU. Another leading player in the facial recognition industry is Affectiva, co-founded in 2009 by Rana el Kaliouby and Rosalind Picard while they studied at MIT's Affective Computing lab, which has now analyzed over 7.5 million faces from 87 countries.

NTechLab definitely has your face if you live in Russia, as the company claimed in 2017 they could track every user on VK, and can search a database of a billion faces in half a second. With 2,000 customers including governments, casinos, and talent agencies in Russia, Turkey, the US, UK, Australia, India, and China, they also surveilled the international crowds at the 2017 FIFA Confederations Cup.

And of course, the worldwide leader in mass surveillance and facial recognition is China. Beijing-based Megvii Technology's Face++ offers a menu of facial analysis services including face search, eye tracking, emotion recognition, skin health analysis, skeleton detection, beauty score, and many, many others.

Emotion recognition with example image by Face++
Beauty score with example image by Face++
Skeleton Detection example by Face++

Between Face++, facial recognition competitor SenseTime, and ByteDance (the owners of TikTok, the viral video app which has been downloaded 800 million times), and the Chinese government's own "Safe Cities" program, which have installed 170 million cameras in Chinese cities and plan to expand to another 400 million by 2020, a very large fraction of the global population already is or will be identified in facial recognition databases with the capacity to detect emotions in real time in the very, very near future.

What we're serving all this face for

Shoshana Zuboff's The Age of Surveillance Capitalism describes how tech companies have manufactured new kinds of capital from every aspect of human existence and society via surveillance technology and big data analysis. This is exactly how all of these facial and emotional recognition companies are turning a profit on harvesting emotional data: every smile, frown, and neutral expression that can be captured, can be monetized.

Some of the applications for emotional recognition data aren't purely profit-driven, and will actually serve humanity: medical applications, for example, which help with identifying recovery from depression, in social robots that assist the elderly, and teaching children on the autism spectrum to identify and mimic emotional social cues.

Affectiva has received a lot of positive attention for its application of emotional recognition in smart cars, monitoring both drivers' awareness for vehicular safety, and the occupants' experience for a more relaxing, fun, or productive ride. Other public health and safety applications such as crowd surveillance in large events and connecting school surveillance systems directly to police departments could be used to prevent or mitigate terror attacks and school shootings.

But the most obvious and loudly touted benefits of emotion recognition are all marketing-related. As Google's Senior VP of Global Marketing, Lorraine Twohill, has said, “If we don’t make you cry, we fail. It’s about emotion.” Hamish Pringle and Peter Field's book Brand Immortality found that advertising campaigns focused on emotional content perform twice as well as those that rationally explain the benefits of a product. It's a marketing hack that no person is immune to, and for tech companies like Facebook and Amazon that are always trying to increase their influence over human behavior, it's priceless interaction data.

In a 2017 interview with O'Reilly, one of Affectiva's co-founders revealed that they work with one third of Fortune 500 companies, and an anecdote reported by Hubspot reveals Affectiva's influence in a specific campaign – this Kellogg's Crunchy Nut Granola commercial was chosen through a series of emotion recognition experiments that revealed the alien produced more audience engagement than other characters:

The alien is cute, but it's a harbinger of a world in which targeted ads could be chosen based not only on your location, income, nationality, gender, and online behavior, but also on your current mood.

The most stunning example of this was the new Piccadilly Lights installed in London in 2017 by a company called Landsec. The new 11 million pixel LED screen uses embedded cameras to determine the make and model of passing cars, as well as the demographic and emotional response data of pedestrians, and responds with targeted ads.

My body, my mood, my rules

It's obvious from the applications of these technologies in massive public venues, in schools, invisibly all over the internet, in the hands of governments and law enforcement, that there is currently no regulation; there is no "opt in" required for the use of data gleaned by looking at your face.

NTechLab's CEO, Mikhail Ivanov, has been quoted by Mashable with the statement: "We don't believe in privacy. We live in a world of CCTV cameras everywhere and internet communications. The concept of privacy applied to the world of our grandfathers, not our world."

This question of privacy, whether the boundaries of the physical body exist at all in the digital economy of surveillance capitalism, is a fraught one that human rights institutions like the ACLU,  Amnesty International, and Privacy International are fighting to redefine and defend.

Emotional analysis algorithms are far from perfect, as previous studies have found difficulties in accurately classifying gender and emotional content on darker skin tones, and can lead to over-policing of black and brown people due to algorithmic bias.

So on one hand, AI are not reliable enough to help police, but on the other, they could become so good at recognizing "genuine" and "false" emotional expressions (i.e. smiling, but without the eyes) that they prevent our ability to lie – even when it is socially acceptable, or beneficial, to do so, undermining freedom of speech and expression.

Meredith Whittaker, from AI Now, a research institute examining the social implications of artificial intelligence, points out that the use of emotion recognition in determining things like students' engagement in the classroom and job interviewers' sincerity or desperation could cause concrete material harm if their findings are inaccurate or deployed unfairly.

Plus, the whole science of emotion has been called into question: is what we think and feel really so easily read on our faces? Are socially unacceptable responses like laughing at dead baby jokes, or crying when the government or your boss believes you should be stoic soon to become punishable offenses?


It's fun to think of a cyberpunk future in which everyone wears funky haircuts and makeup as face detection camouflage and adversarial prints that register them as fleets of cars to computer vision, but without knowing who's watching, where, when, or why, it's harder to react to these market forces trying to shape our behavior into perfect consumers.

It would be difficult to go about daily life never looking directly at a camera or an advertisement, and even this refusal to make eye contact with any potential facial recognition technology would most likely be logged as aberrant behavior and as significant data to be capitalized on.

Plus, given all of the above information, for the majority of the world it's far too late to take back control of our face data. What we're really fighting for is control of our own behavior, with the awareness that we are being watched and manipulated on all sides.

Maybe the next time an advertisement tries to make you laugh, think of something so sad that you make yourself cry. Scowl directly into your webcam for an extra hour every day. Yell at the next electronic display you see on the street that you know they're watching. Grin maniacally into every CCTV camera you can.

And the next time anyone tells you to smile, whether it's an online ad, a man on the street, or the all-seeing AI government entity finally sick of your shit attitude, tell it talk to the hand, because the face ain't listening.