1895-975X For Now We See through an AI Darkly; but Then Face-to-Face: A Brief Survey of Emotion Recognition in Biometric Art Schiller Devon author Department of English and American Studies, University of Vienna Correspondence to: Devon Schiller 19 08 2020 Numer 3 (45) 2020 230 260 Copyright © 2020 2020 <p>Our knowledge about the facial expression of emotion may well be entering an age of scientific revolution. Conceptual models for facial behavior and emotion phenomena appear to be undergoing a paradigm shift brought on at least in part by advances made in facial recognition technology and automated facial expression analysis. And the use of technological labor by corporate, government, and institutional agents for extracting data capital from both the static morphology of the face and dynamic movement of the emotions is accelerating. Through a brief survey, the author seeks to introduce what he terms biometric art, a form of new media art on the cutting-edge between this advanced science and technology about the human face. In the last ten years, an increasing number of media artists in countries across the globe have been creating such biometric artworks. And today, awards, exhibitions, and festivals are starting to be dedicated to this new art form. The author explores the making of this biometric art as a critical practice in which artists investigate the roles played by science and technology in society, experimenting, for example, with Basic Emotions Theory, emotion artificial intelligence, and the Facial Action Coding System. Taking a comprehensive view of art, science, and technology, the author surveys the history of design for biometric art that uses facial recognition and emotion recognition, the individuals who create such art and the institutions that support it, as well as how this biometric art is made and what it is about. By so doing, the author contributes to the history, practice, and theory for the facial expression of emotion, sketching an interdisciplinary area of inquiry for further and future research, with relevance to academicians and creatives alike who question how we think about what we feel.</p> 1 Affectiva, Affectiva, n.d., https://www.affectiva.com/ (accessed: 11.12.2019). 2 American Society for Aesthetic Plastic Surgery (ASAPS), Cosmetic (Aesthetic) Surgery National Data Bank Statistics 2018, https://www.surgery.org/sites/default/files/ASAPS-Stats2018.pdf (accessed: 11.12.2019). 3 Andréasson P., Emotional Empathy, Facial Reactions, and Facial Feedback, “Digital Comprehensive Summaries of Uppsala Dissertations from the Faculty of Social Sciences” 2010, 58, Uppsala Universitet, Uppsala, Sweden, http://www.diva-portal.org/smash/get/diva2:327146/FULLTEXT01.pdf (accessed: 11.12.2019). 4 Animal FACS, Animal Facial Action Coding Systems (FACS), Psychology Department, University of Portsmouth, http://animalfacs.com/index.html (accessed: 11.12.2019). 5 Arthur H., Kittydar: Face Detection for Cats in JavaScript, GitHub, 2012, http://harthur.github.io/kittydar/ (accessed: 11.12.2019). 6 Barrett L.F., How Emotions Are Made: The Secret Life of the Brain, Houghton Mifflin Harcourt, Boston 2017. 7 BioID, BioID Face Database, n.d., https://www.bioid.com/About/BioID-Face-Database (accessed: 11.12.2019). 8 Bismarck J. von, Public Face II, 2010, http://juliusvonbismarck.com/bank/index.php?/projects/public-face-ii/ (accessed: 11.12.2019). 9 Boehner K., DePaula R., Dourish P., Sengers P., How Emotion Is Made and Measured, “International Journal of Human-Computer Studies” 2007, Vol. 65, No. 4, pp. 275–291, DOI: 10.1016/j.ijhcs.2006.11.016. 10 Boys-Stones G., Physiognomy and Ancient Psychological Theory, in: Seeing the Face, Seeing the Soul: Polemon’s Physiognomy from Classical Antiquity to Medieval Islam, ed. S. Swain, Oxford University Press, Oxford 2007, pp. 19–124. 11 Caeiro C., Burrows A., Waller B.M., Development and Application of CatFACS: Are Human Cat Adopters Influenced by Cat Facial Expressions?, “Applied Animal Behaviour Science” 2017, No. 189, pp. 66–78, DOI: 10.1016/j.applanim.2017.01.005. 12 Celeghin A., Diano M., Bagnis A., Viola M., Tamietto M., Basic Emotions in Human Neuroscience: Neuroimaging and Beyond, “Frontiers in Psychology” 2017, Vol. 8, No. 1432, pp. 1–29, DOI: 10.3389/fpsyg.2017.01432. 13 Darwin C., On the Origin of Species by Means of Natural Selection, or the Preservation of Favoured Races in the Struggle for Life, John Murray, London 1859. 14 Darwin C., The Expression of the Emotions in Man and Animals, John Murray, London 1872. 15 Das Gesicht: Bilder, Medien, Formate (The Face: A Search for Clues), ed. S. Weigel, Wallstein Verlag, Göttingen 2017. 16 Davis J.I., Senghas A., Brandt F., Ochsner K.N., The Effects of BOTOX Injections on Emotional Experience, “Emotion” 2010, Vol. 10, No. 3, pp. 433–440, DOI: 10.1037/a0018690. 17 Deleuze G., Guattari F., A Thousand Plateaus: Capitalism and Schizophrenia, transl. B. Massumi, University of Minnesota Press, Minneapolis, MN 1998. Originally published in French as Capitalisme et Schizophrénie 2: Mille Plateaux, Les Éditions de Minuit, Paris 1980. 18 Ekman P., Darwin’s Contributions to Our Understanding of Emotional Expressions, “Philosophical Transactions of the Royal Society B” 2009, No. 364, pp. 3449–3451, DOI: 10.1098/tstb.2009.0189. 19 Ekman P., Methods for Measuring Facial Action, in: Handbook of Methods in Nonverbal Behavior Research, eds. K.R. Scherer, P. Ekman, Cambridge University Press, Cambridge 1982, pp. 45–90. 20 Ekman P., Nonverbal Messages: Cracking the Code – My Life’s Pursuit, Paul Ekman Group, San Francisco 2016. 21 Ekman P., Universals and Cultural Differences in Facial Expressions of Emotions, in: Nebraska Symposium on Motivation, ed. J. Cole, University of Nebraska Press, Lincoln, NB 1972, pp. 207–282. 22 Ekman P., Davidson R.J., Friesen W.V., The Duchenne Smile: Emotional Expression and Brain Physiology II, “Journal of Personality and Social Psychology” 1990, Vol. 58, No. 2, pp. 342–253, DOI: 10.1037/0022-3514.58.2.342. 23 Ekman P., Friesen W.V., Hager J.C., FACS Manual on CD-ROM, Network Information Research Corporation, Salt Lake City, UT 2002. 24 Ekman P., Sorenson E.R., Friesen W.V., Pan-Cultural Elements in Facial Displays of Emotion, “Science” 1969, Vol. 164, No. 3875, pp. 86–88, DOI: 10.1126/science.164.3875.86. 25 El Kaliouby R., Colman C., Girl Decoded: A Scientist’s Quest to Reclaim Our Humanity by Bringing Emotional Intelligence to Technology, Penguin Books, New York 2020. 26 Faceless: Re-Inventing Privacy Through Subversive Media Strategies, eds. B. Doringer, B. Felderer, Edition Angewandte, De Guyter Mouton, University of Applied Arts, Berlin–Vienna 2018. 27 Facial Recognition Exhibition Catalogue, ed. D. Stein, The Wende Museum, Culver City, CA 2016, https://wendemuseum.files.wordpress.com/2016/05/facial-recognition-catalog.pdf (accessed: 11.12.2019). 28 Fedorova K., Tactics of Interfacing: Encoding Affect in Art and Technology, The MIT Press, Cambridge, MA 2020. 29 Flusser V., Into the Universe of Technical Images, transl. N.A. Roth, University of Minnesota Press, Minneapolis, MN 2011. Originally published in German as Ins Universum der technischen Bilder, European Photography, Berlin 1985. 30 Fraunhofer IIS, Press Release: GfK EMO Scan Wins 2012 German Innovation in Market Research Award, 18.07.2012, https://www.iis.fraunhofer.de/content/dam/iis/en/doc/pr/2012/201207112_PI_EMO_Scan_e.pdf (accessed: 11.12.2019). 31 Fraunhofer IIS, SHORE Demo Software, https://www.iis.fraunhofer.de/en/ff/sse/imaging-and-analysis/ils/dl/shore/terms.html (accessed: 11.12.2019). 32 Fraunhofer Institute for Integrated Circuits (IIS), SHORE Face Detection Software, https://www.iis.fraunhofer.de/en/ff/sse/ils/tech/shore-facedetection.html (accessed: 11.12.2019). 33 Fridlund A.J., Human Facial Expression: An Evolutionary View, Academic Press, San Diego 1994. 34 Fridlund A.J., The Behavioral Ecology View of Facial Displays, 25 Years Later, in: The Science of Facial Expression, eds. J.-M. Fernández-Dols, J.A. Russell, Oxford University Press, New York 2017, pp. 77–92. 35 Frijda N.H., The Psychologists’ Point of View, in: The Handbook of Emotions, 3rd Edition, eds. M. Lewis, J.M. Haviland-Jones, L.F. Barrett, The Guilford Press, New York 2008, pp. 68–87. 36 Gao Y., Neutralité: Can’t and Won’t, 2016, http://www.yinggao.ca/interactifs/neutralite--cant-and-wont/ (accessed: 11.12.2019). 37 Gao Y., Profile, http://yinggao.ca/info/profile/ (accessed: 11.12.2019). 38 Garbas J., SHORE Image Analysis: Data-Protection Compliant and Certified, Interview with Jens Garbas, Fraunhofer IIS, 2017, https://www.iis.fraunhofer.de/en/magazin/2017/datenschutz-shore.html (accessed: 11.12.2019). 39 Gates K.A., Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance, New York University Press, New York 2011. 40 Hansen M.B.N., Bodies in Code: Interfaces with Digital Media, Routledge, New York 2006. 41 Hernandez J., Ehsan Hoque M., Mood Meter, 2011, https://moodmeter.media.mit.edu (accessed: 11.12.2019). 42 Hernandez J., Ehsan Hoque M., Drevo W., Picard R.W., Mood Meter: Counting Smiles in the Wild, in: UbiComp ’12: Proceedings of the 2012 Association for Computing Machinery (ACM) Conference on Ubiquitous Computing, eds. A.K. Dey, H.H. Chu, G. Hays, Association for Computing Machinery, New York 2012, pp. 301–310, DOI: 10.1145/2370216.2370264. 43 iMotions Biometric Research Platform, Affectiva, https://imotions.com/affectiva-requestdemo (accessed: 11.12.2019). 44 Jones C., What You Staring At? Ying Gao’s Gaze-Activated Dresses, “Vice”, 19.06.2013, https://www.vice.com/en_us/article/ez5g47/ying-gaos-gaze-activated-dresses (accessed: 11.12.2019). 45 Kaczor E., Julius von Bismarck und der Smiley über Berlin, “ART Berlin”, 2.06.2012, http://www.artberlin.de/kuenstler/julius-von-bismarck-smiley (accessed: 11.12.2019). 46 Lee-Morrison L., Portraits of Automated Facial Recognition: On Machinic Ways of Seeing the Face (Image), Transcript Verlag, Bielefeld 2019. 47 Lisetti C.L., Schiano D.J., Automatic Facial Expression Interpretation: Where Human-Computer Interaction, Artificial Intelligence and Cognitive Science Intersect, “Pragmatics and Cognition: Special Issue on Facial Information Processing: A Multidisciplinary Perspective” 2000, Vol. 8, No. 1, pp. 185–235, DOI: 10.1075/pc.8.1.09lis. 48 Loijens L., Krips O., FaceReader Methodology Note, white paper, Noldus Information Technology, 2019, https://www.noldus.com/facereader/resources (accessed: 11.12.2019). 49 Lyons M.J., Budynek J., Akamatsu S., Automatic Classification of Single Facial Images, “IEEE Transactions on Pattern Analysis and Machine Intelligence” 1999, Vol. 21, No. 12, pp. 1357−1362, DOI: 10.1109/34.817413. 50 Nake F., Surface, Interface, Subface: Three Cases of Interaction and One Concept, in: Paradoxes of Interactivity: Perspectives for Media Theory, Human-Computer Interaction, and Artistic Investigations, eds. U. Seifert, J. Hyun Kim, A. Moore, Transcript, Bielefeld 2008, pp. 92–109, DOI: 10.25969/mediarep/2719. 51 Noldus, FaceReader, n.d., https://www.noldus.com/facereader (accessed: 11.12.2019). 52 Oxford Handbook of Face Perception, eds. A.J. Calder, G. Rhodes, M.H. Johnson, J.V. Haxby, Oxford University Press, Oxford–New York 2011. 53 Ożóg M., Życie w krzemowej klatce. Sztuka nowych mediów jako krytyczna analiza praktyk cyfrowego nadzoru (Life in a Silicon Cage: New Media Art as a Critical Analysis of Digital Surveillance Practices), Wydawnictwo Uniwersytetu Łódzkiego, Łódź 2018. 54 Pantic M., Machine Analysis of Facial Behavior: Naturalistic and Dynamic Behavior, “Philosophical Transactions of The Royal Society B” 2009, Vol. 364, pp. 3505–3513, DOI:10.1098/rstb.2009.0235. 55 Pantic M., Bartlett M.S., Machine Analysis of Facial Expressions, in: Face Recognition, eds. K. Delac, M. Grgic, I-Tech, Vienna 2007, pp. 377–416. 56 Phillips P.J., Moon H., Rizvi S.A., Rauss P.J., The FERET Evaluation Methodology for Face-Recognition Algorithms, National Institute of Standards and Technology (NIST) Interagency/Internal Report (NISTIR) 6264, 1998, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=900863 (accessed: 11.12.2019). 57 Picard R.W., Affective Computing, The MIT Press, Cambridge, MA 1997. 58 Picard R.W., Affective Computing, Tech. Report: 321, The MIT Press: MIT Media Lab, Cambridge, MA 1995. 59 Picard R.W., Toward Machines with Emotional Intelligence, in: Proceedings of the First International Conference on Informatics in Control, Automation and Robotics, ICINCO 2004, Setúbal, Portugal, August 25–28, 2004, pp. 1–22, DOI: 10.1093/acprof:oso/9780195181890.003.0016. 60 Press S., Dresses that React to the Emotions of Viewers, “Ignant” 2016, https://www.ignant.com/2017/02/03/dresses-that-react-to-the-emotions-of-viewers/ (accessed: 11.12.2019). 61 Rajewsky I.O., Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality, “Intermédialités” 2005, No. 6, pp. 43–64, DOI: 10.7202/1005505ar. 62 Rinn W.E., The Neuropsychology of Facial Expression: A Review of the Neurological and Psychological Mechanisms for Producing Facial Expressions, “Psychological Bulletin” 1984, No. 95, pp. 52–77, DOI: 10.1037/0033-2909.95.1.52. 63 Rizzolatti G., Sinigaglia C., Mirrors in the Brain: How Our Minds Share Actions and Emotions, transl. F. Anderson, Oxford University Press, Oxford 2008. Originally published in Italian as Specchi nel cervello: Come comprendiamo gli altri dall’interno, Raffaello Cortina Editore, Milan 2006. 64 Rosenberg E., Introduction: The Study of Spontaneous Facial Expressions in Psychology, in: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), 2nd Edition, eds. P. Ekman, E. Rosenberg, Oxford University Press, Oxford 2005, pp. 3–18 (1st edition 1998). 65 Ruf T., Ernst A., Küblbeck C., Face Detection with the Sophisticated High-speed Object Recognition Engine (SHORE), in: Microelectronic Systems: Circuits, Systems and Applications, eds. A. Heuberger, G. Elst, R. Hanke, Springer Verlag, Heidelberg 2011, pp. 243-252, DOI: 10.1007/978-3-642-23070-7-23. 66 Russell J.A., A Circumplex Model of Affect, “Journal of Personality and Social Psychology” 1980, Vol. 39, No. 6, pp. 1161–1178, DOI: 10.1037/h0077714. 67 Schiller D., Eco’s Face Drawing and the Continuum of Emoticon Articulation, in: Umberto Eco in His Own Words, eds. T. Thellefsen, B. Sørensen, De Gruyter Mouton, Berlin 2017, pp. 57–70, DOI: 10.1515/9781501507144-009. 68 Schiller D., Face, A Keyword Story: The Archiving Vocabulary for Facial Expression in the German Imaginary from Printed Text to Digital Image, in: Digital. Media and Textuality: From Creation to Archiving, ed. D. Côrtes Maduro, Transcript Verlag, Bielefeld 2017, pp. 207–252, DOI: 10.14361/9783839440919-013. 69 Schiller D., Greening Face: How Color Makes Facial Expression Sensible, from Pre-Christian Architectural Spaces to Post-Digital Smart Environments, “Arts &amp; Cultural Studies Review” 2018, Vol. 4, No. 38, pp. 493–534, DOI: 10.4467/20843360PK.18.026.10364. 70 Schiller D., On the Basis of Face: Biometric Art as Critical Practice, its History and Politics, Institute of Network Cultures, Amsterdam 2020. 71 Seung-Back S., Yong-Hun K., Cat or Human, 2013, http://ssbkyh.com/works/cat_human/ (accessed: 11.12.2019). 72 Sick-Leitner M., Featured Artists 2014 Ars Electronica, Ars Electronica Blog, 18.09.2014, https://ars.electronica.art/aeblog/en/2014/08/18/featured-artists-ars-electronic-2014/ (accessed: 11.12.2019). 73 Twardoch E., Are There Stories Hidden Behind Hormonal Spaces and X-Ray Photographs? Around Narratives in Biometrics-based New Media Art, in: The Practice of Narrative: Storytelling in a Global Context, eds. A. Penjak, M. Heitkemper-Yates, Brill, Leiden 2016, pp. 113–122, DOI: 10.1163/9781848883802_012. 74 Ven R. van de, Choose How You Feel; You Have Seven Options, Institute of Network Cultures (INC) Longform, Amsterdam 2017, http://networkcultures.org/longform/2017/01/25/choose-how-you-feel-you-have-seven-options/ (accessed: 11.12.2019). 75 Ven R. van de, Emotion Hero: Android App, Google Play Store, 2017, https://play.google.com/store/apps/details?id=com.rubenvandeven.emotion_hero (accessed: 11.12.2019). 76 Ven R. van de, Emotion Hero: Game Demo, Vimeo video, 02:08, 2017, https://vimeo.com/191263327 (accessed: 11.12.2019). 77 Ven R. van de, Emotion Hero, Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&amp;C ’17), 2017, pp. 422–423, DOI: 10.1145/3059454.3059490/. 78 VicarVision, FaceReader, n.d., http://www.vicarvision.nl/products/facereader/ (accessed: 11.12.2019). 79 Vries P. de, Schinkel W., Algorithmic Anxiety: Masks and Camouflage in Artistic Imaginaries of Facial Recognition Algorithms, “Big Data &amp; Society” 2019, Vol. 6, No. 1, pp. 1–12, DOI: 10.1177/2053951719851532. 80 Wagner P., Face Recognition with OpenCV2, Open Source Computer Vision (OpenCV), 2012 https://docs.opencv.org/3.3.1/da/d60/tutorial_face_main.html (accessed: 11.12.2019). 81 Wang C., Frontal Face Images, CMU Image Data Base: Face, 2000, http://www.cs.cmu.edu/afs/cs/project/vision/vasc/idb/www/html_permanent/index.html (accessed: 11.12.2019). 82 Whittaker M., Crawford K., Dobbe R., Fried G., Kaziunas E., Mathur V., West S.M., Richardson R., Schultz J., Schwartz O., AI Now Report 2018, AI Now Institute and New York University, New York 2018. 83 Wichers M., Face Design, 2019, https://www.projectfacedesign.com (accessed: 11.12.2019). 84 Wichers M., How Botulinum Toxin Affects Facial Expression, Noldus Behavioral Research Blog, 25.02.2019, https://www.noldus.com/blog/how-botulinum-toxin-affects-facial-expression (accessed: 11.12.2019). 85 Wilhelmer R., Fühl-O-Meter, 2010, http://richardwilhelmer.com/projects/fuhl-o-meter (accessed: 11.12.2019). 86 Wilkins A., Making Faces: The Evolutionary Origins of the Human Face, Harvard University Press, Cambridge, MA 2017. 87 Wirth S., Faszination Gesicht: Was unsere Mimik alles zeigt (Fascination Face: What Our Expressions Show), “Vögelekultur Bulletin” 2019, No. 107, Vögele Culture Center, Pfáffikon, Switzerland. 88 Zhang W., Sun J., Tang X., Cat Head Detection: How to Effectively Exploit Shape and Texture Features, in: Computer Vision – ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12–18, 2008, Proceedings, Part IV, eds. D. Forsyth, P. Torr, A. Zisserman, Springer Verlag, Berlin–Heidelberg 2008, pp. 802–816, DOI: 10.1007/978-3-540-88693-8-59.