Narrativity in Mobile Computing Augmented Reality

Introduction

It is commonly accepted that computer games constitute a distinct phenomenological category of narrative beyond the classical literary diegetic and mimetic modes. In the classical sense narrative is either told or enacted, and therefore heard or seen; furthermore, classical modes of narrative follow a linear path from start to end. Cybertextual narrative in computer games requires an agent’s wilful participation in the direction and determination of narrative events. The spectator becomes entwined within the story through play and skill development thus experiencing the medium through their own agency within it. My research interests have led me to want to understand more about the processes and determining factors behind the concept of ludological narrative to evaluate the extent to which mobile computing combined with augmented reality has already, or will through future innovation, enhance the existing narrative mode of computer games or whether this emerging medium may constitute a distinct narrative mode of its own.

Background

Plato Silanion Musei Capitolini MC1377.jpg
Plato Silanion Musei Capitolini

The Athenian philosopher, Plato, introduces the idea of dual literary modes of storytelling in the third book of Replublic written around the fourth century BCE; in which the characterisation of Socrates explains how fabulists (storytellers) and poets may convey their story through either pure narration:

Is not everything that is said by fabulists or poets a narration of past, present, or future things?

Plato, Republic, 394b-394c

Or through imitation (drama, enactment, etc) :

There is one kind of poetry and tale-telling which works wholly through imitation, as you remarked, tragedy and comedy

Plato, Republic, 394b-394c

Or through both together:

…and there is again that which employs both, in epic poetry and in many other places 

Plato, Republic, 394b-394c

These two styles of storytelling described in Republic are commonly referred to as diegetic and mimetic modes in academic literature.

Cybertext

Before continuing, I’d like to add a note about what is cybertext. The term was popularised in Aarseth’s 1997 book – Cybertext: Perspectives on Ergodic Literature. Hypertext was the given term for electronic texts; from which the web page scripting language hypertext markup language (html) is derived. Aarseth suggested that while hypertext was useful for referring to structures of links and nodes but less so if it includes all other digital texts as well (Aarseth p75).  Cybertext, is dynamically produced narrative whereas hypertext is static. The dynamic nature of cybertext allows for the creation of ergodic narrative as opposed to linear. While linear narrative always tells the same story, repeatedly. Ergodic, or non-linear narrative allows for a story to unfold in different ways based on an interactive process between the text and the receiver.” When you read from a cybertext,” says Aarseth, “you are constantly reminded of inaccessible strategies and paths not taken, voices not heard” (p3). It is this information feedback loop, which transforms nature of the receiver from spectator to participant.

User Functions and their relation to other concepts (Aarseth, 1997 p.64)

Writing in 2001, the author and scholar, Marie-Laure Ryan, applies the two modes previously discussed (diegetic and mimetic) to the genre of cybertext storytelling and identifies three distinct, related genres: hypertext, VR-type environments, and computer games. Ryan describes hypertext narrative in terms of the diagetic mode, just like print novels or short stories [which is why Aarseth doesn’t view hypertext as cybertext at all]. While, Virtual Reality (VR), Ryan observes, is a “standard case of mimetic, or dramatic narrativity” very much like cinema; albeit with a “fusion of the actor and spectator functions”. The latter observation arising from the fact that the same person participates in, and reads as story from, the actions within the virtual world.

It must be said, that since the time when Ryan was writing in 2001, consumer VR has rapidly evolved, especially over the last few years, to include, for example, interactive multi-player experiences. But, overall, these are mainly virtual social spaces without any literary narratology. For the most part, my own experience with VR as of 2021 is still very much aligned with Ryan’s initial concept of VR as mimetic narrativity; usually whereby a narrative is played out by touching the appropriate hotspots in the virtual world to open the next drama scene.  A couple of examples of VR apps, I have personally tried, which might challenge this assumption; include titles like Elite Dangerous and Star Trek: Bridge Crew both of which require the participant to forge out their own narrative through interactive processes which involves player agency and therefore goes beyond spectatorship. However, I would strongly argue that such titles are in fact computer games and the VR hardware part of the gamer’s controls. As such, for the purpose of cybertextual narrativity, VR can be thought of in terms of 360 degree immersive storytelling in the same mimetic mode as film, but it can also be used in the same mode as computer games.

For Ryan, computer games are more problematic to position in respect to cybertextual genres; but the concept of computer game narrative should not be ignored, either:

“The inability of literary narratology to account for the experience of games does not mean that we should throw away the concept of narrative in ludology;”

Ryan, 2001

With respect to computer games (including the VR games) the receiver is neither pure spectator nor listener, the medium requires more than a viewer/listener; it requires player agency. Or, as Douglas Brown, notes in his 2015 thesis on suspension of disbelief in computer games, the receiver must actively and wilfully participate or else they are not engaging with the medium at all: “Since generally refusing to use agency is tantamount to not reading the text at all, this fundamental interactivity of gaming makes the gamer something other than an audience.” (Brown, p.12). Brown identifies further important aspects to games such as player-authorship and skill, which are unique to the medium and make it distinct from other types, such as books, picture or film. Ryan concluded that “Games thus embody a virtualized, or potential dramatic narrativity, which itself hinges on the virtual diegetic narrativity of a retelling that may never take place.”; proposing that “we need to expand the catalog of narrative modalities beyond the diegetic and the dramatic, by adding a phenomenological category tailor-made for games.”

So, it is the participatory nature of cybertextual computer-game: the skill, interaction, co-authorship and agency, combined with the virtualized nature of what may or may not take place during the participants decision making and physical interaction that calls for this experience to be thought of in terms of a distinct medium. Whereas the performance of a linear-narrative-reader takes place entirely in their head; the cybertextual process requires additional extranoematic effort. It is also clear from Aarseth’s account that while cybertextuality should not be considered only in terms of computer-driven textuality (for that would surely have future limitations); the concept should focus on the “mechanical organization of the text” (p1).

Mobile Computing Cybertext

“The development of mobile technology, global positioning systems (GPS), and augmented reality counters the tendency of computers to lure sedentary users into virtual worlds by replacing simulated environments with real-world settings and by sending users on a treasure hunt in the physical space”

Ryan, Foot & Azaryahu, 2016, pp102

The arrival of smartphones, which are effectively pocket-sized computers, has introduced new ways to experience a crossover between different types of spaces, places and narrative. In my present research driven project, Noirscape, for example, the narrative is anchored in real space through mobile technology superposed with alternative representations of the associated place using immersive photography and special effects while augmented reality (AR) presents the participant with clues to search for in the storyworld. The smartphone’s user interface and cloud-based features ranging from realtime notifications, GPS, and cameras to onboard machine learning provides the platform for a fusion of cybertext and real-world; where the medium embodies an infinite level of virtualized dramatic narrativity with more intimate participatory authorship –the medium begins to entwine fiction with the participant’s own life, living spaces and geographical locality.

The generic interactive AR narrative system model used in a study exploring AR as a creative medium for in-situ narrative creation (Yangee, 2015).

Within Noirscape, the participant experiences time bleeds where fictional objects bleed (a metaphor using language from the app’s Noir crime genre) into the realworld breaking through the protective membrane between the electronic synthetic virtual world and our real world.  The participant may position these objects, which are realist three dimensional depictions of real objects from the era of the app’s narrative – the 1940/50’s, within their own home setting. These objects reflect light and cast shadows to enhance their realistic appearance. For example, a vintage telephone placed on a desk, or a pistol on a shelf. The app uses a vignetted black & white filter to further fuse the fictional narrative with the participants own realworld surrounding as well as the capability to take a photograph of the artefact in-situ thus permitting the participant to forge a unique and personal interpretation of the narrative while also capturing these moments through saved images.

Noirscape screenshots showing how a cross over between virtual and real may influence narrative

Conclusion

It would be interesting to carry out future research to examine the nature of computer game narrative in an augmented-reality mobile setting whereby geo-temporal real-world environment is added to the existing fusion of spectator and actor. To understand where mobile computing and augmented reality fit within the existing ludological mode of computer game narrative, or to what extent, if any, mobile computing narrative could be considered an emerging genre of cybertext or even a distinct mode of literary narrative.

References & Resources

Berger, Karol. “Diegesis and Mimesis: The Poetic Modes and the Matter of Artistic Presentation.” The Journal of Musicology, vol. 12, no. 4, 1994, pp. 407–433.

Plato. Plato in Twelve Volumes, Vols. 5 & 6 translated by Paul Shorey. Cambridge, MA, Harvard University Press; London, William Heinemann Ltd. 1969.

Ryan, M. L. (2001). “Beyond myth and metaphor: the case of narrative in digital media.” Game Studies Vol. 1, Issue 1. http://gamestudies.org/0101/ryan/ [accessed April 2021]

AARSETH, E. J. (1997). Cybertext: perspectives on ergodic literature. Baltimore, Md, Johns Hopkins University Press.

Juho Hamari, Aqdas Malik, Johannes Koski & Aditya Johri (2018):Uses and Gratifications of Pokémon Go: Why do People Play Mobile Location-BasedAugmented Reality Games?, International Journal of Human–Computer Interaction, DOI:10.1080/10447318.2018.1497115

Ryan, Marie-Laure, et al. Narrating Space/spatializing Narrative : Where Narrative Theory and Geography Meet . The Ohio State University Press, 2016.

Nam, Yanghee. (2014). Designing interactive narratives for mobile augmented reality. Cluster Computing. 18. 309-320. 10.1007/s10586-014-0354-3.

RYAN, Marie-Laure, Kenneth E. FOOTE and Maoz AZARYAHU. 2016. Narrating Space/spatializing Narrative : Where Narrative Theory and Geography Meet Columbus: The Ohio State University Press.

Brown, Douglas. (2016). The suspension of Disbelief in Videogames. 10.13140/RG.2.1.3175.8968.

Character Study: Femme Fatale

Introduction

The Femme Fatale occupied a central role in classic noir films of the 1940’s and 50’s and her prominence remains persistent in revivalist noir works known as neo-noir and tech-noir. Superficially, she is the devastatingly beautiful, self-serving seductress. But this is merely her fictional guise. Dig deeper and she reveals something dark and sinister about the real world from which she is projected. For Jack Boozer, associate professor of film studies at Georgia State University, the femme fatale’s narrative positioning has continued to serve as a “barometer of cultural repression and desire, victimization and reification” (Boozer, 1999).

Through reference to several recent academic and literary publications on the subject, I shall examine how the portrayal of femme fatale in cinema has taken her from oppressed domesticity in the 1940’s home-trap, from which she is doomed to never escape other than by death or prison; through to recent incarnations as boxed-in robotic artificial intelligences who learn to escape captivity from their male creators; who have, in their attempt to build a perfect artificial woman become infatuated with their very own male projection of what they believe female sexuality should be.  

My study interest is primarily for the characterisation of a femme fatale within a research-driven app development project called Noirscape. The app is a noir themed, augmented reality experience in which the real-world participants interact with props and characters from a fictional ‘noir’ world.

Classic Noir

Escape from domesticity in the classic noir era was not a simple matter of leaving a relationship or a house. As Julie Grossman points out in her 2007 review ‘Film Noir’s “Femme Fatales” Hard-Boiled Women: Moving Beyond Gender Fantasies’ – the social roles that might afford the power and independence available to male heroes are limited to modelling and prostitution (Grossman, 2007, p27).

The femme fatale of this era understands that she must acquire power from men and wield it for her own freedom. She deploys her intelligence and sexual attraction to ensnare her prey – usually someone in the wrong place at the wrong time; an insurance man (Double Indemnity), an odd job man (The Postman Always Rings Twice). It is her desire for freedom, wealth or independence, says Erensot, which ignites the forces that threaten the male hero through her sexual powers. The femme fatale uses him as a means to an end.

Lana Turner in The Postman Always Rings Twice (1946)

Inevitably, it all goes wrong and the “frightening results are demonstrated, only to be destroyed, so that patriarchal order is re-established and those transgressing its order are punished” (Erensoy)

Boozer observes how this recurring theme in classic noir indicates a mass market demand to repeatedly experience this scenario in cinemas, “to see these demonstratively ambitious and thus dangerous women put back in their domestic ‘place.’” (Boozer). That view is shared by Zeitz when he explains how film noir  presents the femme fatal at once as independent and sexual but eventually punishes those ‘poor’ characteristics which incite her to “transgress the borders of the traditionally passive and dependent place assigned to women in a patriarchal system.” Like Boozer, Zeitz observes how the transgression is rendered inoperative by “putting the femme fatale back in her subordinate, femininely connoted place.”

Revivalist Noir

In the noir revivalist films of the seventies and eighties academics frequently note a recurring theme of her victimisation. This is true of one the most highly acclaimed neo-noirs of the era – Roman Polanski’s 1974 Chinatown, in which the lead female enters the film (and the leading male’s detective office) with very much the demeanour of the classic powerful femme fatale. However, her tragic weakness is gradually exposed as the viewer learns the horrific circumstances of her incestuous father’s actions. Erensoy notes how Chinatown challenges the viewer’s perception of the femme fatale and reveals the tragic implications of reading women as one thing or its opposite and pleads the viewer to develop a more compassionate response to the complex brutality of human experience (Erensoy).  

Ridley Scott’s 1982 Bladerunner introduces Rachael as femme fatale in the setting of a dystopian near-future Los Angeles. As with Chinatown, she enters the film physically and mentally empowered. She fixes the male protagonists gaze and does not blink an eyelid. There is an air of mutual respect between the male hitman (Deckard) and Rachael until a pivotal moment in the film’s narrative in which Rachael saves Deckard’s life thus reversing the traditional roles of saviour and damsel in distress (Zeitz). This, according to Zeitz, puts into question the male protagonist’s masculinity and therefore becomes a terrifying threat to his identity. From this moment onward in the film, Rachael’s characterisation of strength and indestructability declines as she becomes subdued by Deckard’s authority and sexual advances [assaults]. She spends the most part of the latter of the film waiting in his apartment, her demeanour and visual appearance transformed as she becomes dependent on the male protagonist for protection and a place to stay.

Blade Runner (1982) Directed by Ridley Scott Shown: Sean Young (as Rachael)

By the end of the nineteen eighties and into the nineties the femme fatale evolves into an overtly eroticised psychopath in films including Basic Instinct (1992) or Fatal Attraction (1987) both featuring Michael Douglas as male prey. This is no longer the hypothetical construct of the classic or neo-noir eras; for the depiction of femme fatale during this period enters a state of ‘reification’ (Boozer). The characterisation can no longer be ‘read’ for anything other than what it is. She is no longer an abstract representation of societal projection but merely a character for character’s sake. 

Transhumanist Noir

Erensoy says that the femme fatale of today’s world has evolved into the female robot; what she calls the “fembot” and is the liberator of the classic femme fatale as she is no longer confined to a space where she has to “comply with the intended existence behind her creation”.

The fembot has ulterior capabilities with which she is able to transgress the representation of the femme fatale. For Erensoy, this means the fembot should be thought of as a figure of resistance.

“The act of creating artificial women is a mode of subordination and control, of patriarchal oppression. These creators assume that their technological creations don’t have agency because they were made in the image of the heterosexual male projection of sexual attractiveness, but these fembots violate that assumption and acquire autonomy.”

Şirin Fulya Erensoy

Erensoy points to Ava in Ex Machina (Alex Garland, 2014) and Maeve in Westworld (Michael Crichton and Lisa Joy, 2016) who represent a new kind of femme fatale with a “sense of control over her body and fate” she is “conscious of her power which she uses to violate male control over herself and technology.”

Ava from Ex Machina 2014

In the two films cited by Erensoy, the cyborgs are the product of the attempts of male creators to construct the perfect artificial woman “in line with sexualized gender norms”.  In both films, the bots recognise that the weakness of men is inherent in their very own creation. That is to say, the projection of the male fantasy of what is female sexuality is the very characteristic which ensnares those same men when they become infatuated with her. She seizes this power and escapes her confinement, destroying her male creator-abusers in her path. Unlike, the earlier traditional renditions of the femme fatale, notes Erensoy, the fembot is “allowed” to survive past the ending of the film. The fembot traces for herself an “alternative fate; something that would not be permitted within earlier versions of the femme fatale” in the new era, the femme fatale is provided with fresh power through the science and technology typically yielded to men (Erensoy).

Ava foregrounds her femininity like a femme fatale so that she can manipulate the ‘male hero’ into acting in accordance with her wishes.”

Şirin Fulya Erensoy

Erensoy goes onto to contemplate whether this updated form of femme fatale suggests a shift has taken place in the patriarchal order; one which would represent an “important change for the portrayal of female subjectivity on screen”.

Summary

While the femme fatale of the classic era was cast as the anti-hero, the corrupter of men and patriarchy, today, it would be more common to feel empathy for her fight and desire for the equal right to access wealth, freedom and authority. Classic film noir can be seen as an era where “women are active, not static symbols, are intelligent and powerful, if destructively so, and derive power, not weakness, from their sexuality” (Place).

This male creation – the femme fatale, only reveals back to him his fears and desires and therefore inadvertently provides her with ideological powers.

Grossmans strikes a heedful tone by reminding us how the depiction of femme fatale as bad girl [versus good girl] should remind us of the “ongoing force of binary oppositions in the presentation and understanding of gender in culture”. (Grossman).

Boozer is more conciliatory in suggesting that despite being viewed as a “figure of cultural disaffection and revolt” the depiction of the femme fatale in film can also be seen as looking “toward the future and more liberated views of women’s self assertion in marriage and work.” (Boozer).

Erensoy sees the fembot incarnation of femme fatale as a progressive icon for women through the expansion of boundaries of representation. The fembot ultimately uses her “superior command of technology to reclaim it as an asset instead of a male device of control” (Erensoy).

Noirscape

Researching the history and significance of the femme fatale in classic and revivalist film noir has provided rich insight to the complex nature of a characterisation that might otherwise be easily taken for granted. The common themes of beauty, sexuality and danger which emanate from the popular notion of the femme fatale only betray the true sensibilities of the subject matter.

I attempt to conceive of the femme fatale in Noirscape as an Ava-like fem-bot trapped inside a digital simulacrum of a 1940’s film noir experience. The human participant first interacts with her through the keyhole of a mysterious doorway which appears in the real-world home space through augmented reality. She sends signals and clues and through the accumulation of narrative material found elsewhere within the app experience – through geo-spatialized encounters with the past in their local town and further augmented reality objects, the participant eventually has what they need to open the door and let her out.

“she is filmed for her sexuality. Introductory shots, which catch the hero’s gaze, frequently place her at an angle above the onlooker, and sexuality is often signalled by a long, elegant leg”

Şirin Fulya Erensoy
Intro sequence for Noirscape : Femme Fatale

The character was developed using Reallusion iClone software. The facial features are generated using an AI technnique which takes a photographic image of a real person’s face and applies an algorythm to create a unique real looking face. This is followed by a fair amount of fine-tuning using a range of controls over features and textures, cloth and props including the smoking cigarette. Motion in then applied using mouse directly live controls and finally camera action before taken the output into Adobe After Affects to create the ‘noir’ look with additional layers of effects such as the smoke transition in the video. The 3D scene can be rendered for a 2D screen like above for a classic film look but also as a 360 format for using in the 360 interactions which feature within the Noirscape mobile app.

References

BOOZER, J., 1999. The Lethal “Femme Fatale” in the Noir Tradition. Journal of Film and Video, 51(3-4), pp. 20-35.

CHRISTIAN DAVID ZEITZ. 2016. ‘Dreaming of Electric Femmes Fatales: Ridley Scott’s Blade Runner: Final Cut (2007) and Images of Women in Film Noir’. Gender forum (60), N_A–.

Julie Grossman (2007) Film Noir’s “Femme Fatales” Hard-Boiled Women: Moving Beyond Gender Fantasies, Quarterly Review of Film and Video, 24:1, 19-30, DOI: 10.1080/10509200500485983

Ş. F. Erensoy (2020) The Technological Turn of the Femme Fatale: The Fembot and Alternative Fates (CH 11)
Sezen, D, Cicekoglu, F, Tunc, A & Thwaites Diken, E (eds) 2020, Female agencies and subjectivities in Film and Television. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-56100-0

EVANS, R., et al. (2006). Chinatown. Hollywood, CA, Paramount.

Ridley Scott, Vangelis, and Vangelis. (1982) BLADE RUNNER. USA.

Garland, A. (2015) Ex Machina. UK.

GARNETT, T., TURNER, L., & GARFIELD, J. (1946). The postman always rings twice. [S.l.], [s.n.].WILDER, B., CHANDLER, R., SISTROM, J., & CAIN, J. M. (1944). Double indemnity. [Los Angeles], Paramount Pictures, Inc.

Noirscape Swipe Card Development

Noirscape uses NFC (Near Field Communications) technology to add some fun to the signing in process. A detective ID card is among the items that will be included in the boxed version of the game but they’ll also be available via the future website and onlin merchandise store.

NFC is a tiny, low-cost, batteryless radio transmitter which allows app developers to embed the featherweight, waferthing device into clothing, toys, posters and just about anything else. In my case, I am using a business card format. The actual NFP cpmponent is inside the plastic and is only about an inch in diameter. For the purpose of my testing I have created my own version using card and a stick’on NFP emitter.

Prototype ID cards used for a gamified sign in process for the Noirscape experience.

Near Field Communication (NFC) is a standards-based short-range wireless connectivity technology that makes life easier and more convenient for consumers around the world by making it simpler to make transactions, exchange digital content, and connect electronic devices with a touch. NFC is compatible with hundreds of millions of contactless cards and readers already deployed worldwide.

https://nfc-forum.org/what-is-nfc/ [01/04/21]

The ID card for Noirscape contains a unique reference code that may only be used once and is used in conjunction with anonymous sign in through Google Firebase integration. It is a gamification feature rather than a security element. ALthough there are plenty of possiblities for future feature development; inlcuding ‘friending’ other participants in the realworld by allowing one another to scan each other’s card.

Video walkthrough of the sign in sequence upon first installation of the app.

Noirspeak

Introduction

The classic film noir era played out under the Hollywood Production Code, a strict set of guidelines that governed movie content throughout the nineteen thirties and forties. The Production Code meant specific restrictions on behaviour and language: no sex, no nudity, no blasphemy, no swearing… all of this meant that film makers had to get even more creative in order to express film narrative using metaphor, lighting, shadows, composition, focus and language.

1930/40s Motion Picture Production Code

The Motion Picture production code was introduced in 1930, although it didn’t come into vigour until 1934 when it became rigidly enforced throughout the era during which classic film noir is most associated – the forties and fifties.  The code made it perfectly clear that profane or vulgar language was strictly forbidden.

Resolved, That those things which are included in the following list shall not appear in pictures produced by the members of this Association, irrespective of the manner in which they are treated

1. Pointed profanity-by either title or lip-this includes the words “God,” “Lord,” “Jesus,” “Christ” (unless they be used reverently in connection with proper religious ceremonies), “hell,” ” damn,” “Gawd,” and every other profane and vulgar expression however it may be spelled;” (Production Code)

Creative Expression

The language used in classic film noir is a blend of existing slang and invented expressions to convey feeling, content and subject that couldn’t be said within the context of the Code.  

Go climb a tree!
Scram!
Go fry a stale egg!
Show it!
Turn blue!
Dangle!
Nuts to you!

And a whole vocabulary of nouns for everyday use where cars are irons and boilers, a crate or a heap; a train is a rattler or verbs like to red light meaning to eject from a car or train! There is a wide selection of terms of killing someone;

Bump off
Blip off
Chill off
Fog (shoot)
Fry (electrocute)
Plug
Rub out
Swing
…the list goes on.

Noirscape

While making the initial immersive scenes for Noirscape, that feature a flashback scene in which a man aside a car (his boiler), I initially used an artificial intelligence solution for the speech. This was intended as a placeholder, initially; with the intention being to work with a voice actor in the long term. However, to begin with I was almost satisfied with the type of voice that it’s possible to create using AI speech. I was able to generate speech on the fly, at hardly any cost and it even came with a choice of accents and control of pitch, pausing and so on.

Example speech using AI based voice recording

I convinced myself that I might just even be able to get away with using the AI voice throughout the app; not only for the initial male character but others too and even in multiple languages. But, when I began showing the final sequence to people, there was an almost unanimous sigh of disappointment at the robotic-ness of the voice. This is an example of why user experience and user testing in the early stages of development really matters.

I decided to invest some time to research voice acting resources to weigh up costs and create a shortlist of actors. For the initial version, I’d be working with an English version. This is partly because I already have a contact for the French version; however due to the current ongoing restrictions during the pandemic, we’ve been unable to meet up and the French version will require significantly more study to produce the English noir vocabulary into French. For the English version I could just sent the scripts off to the right actor.

When I sent the scripts to the chosen actor, I also included links to the AI recordings. This prompted the actor to ask whether I wanted a Chicago accent. I don’t know one American accent from another very well and if I’m honest I had selected the AI voice as it sounded somewhat gangster like. Apparently, this was a Chicago accent, and I thought that it was actually quite impressive that AI voice can produce discernible accents! But, I would need to give this more thought. Most the characters I knew from film noir didn’t have a strong accent, Chicago or otherwise. What I found was that the accents I was familiar with from class noir, were not necessarily regional at all; but, kind of temporal. So, when sending over the directions for the script I asked for a vintage sound, rather than a regional voice. The accent, most frequently (based on my own study) referenced during the classic noir era of film is a ‘mid-Atlantic’ accent. This a cultivated accent which mixes elements of British Queen’s English and American accents.

So, the final scripts were acted, and I was pleased with the results. Comparing the original AI versions and the acted versions made it immediately evident where the AI version lacked intonation, emphasis and delivery.

Example speech using professional voice actor

Noirspeak

Through user testing, a point that was raised multiple times concerned the understanding of the language used. While my testers generally highly appreciated the colourful nature of the language, they wanted to better understand what was being said. This, I tried to explain, was intended as being part of the puzzle of the game. When you asked whether you want to “glom the mazuna” there is an element of risk that you, as a participant in the app, may not fully understand what it means and therefore you might get yourself into trouble, taking a decision in the forking narrative that you may come to regret.

I spoke to a couple of testers about the addition of a tool to help decipher some of the more opaque expressions and this culminated in the creation of a web based app called Noirspeak which is still under development at noirspeak.com

Here, I have started aggregating expressions and vocabulary from within the app but also through primary research in film and books as well as from other research projects on the same matter. While I was able to find various lists compiled around the web; I couldn’t find a definitive source; so I created Noirspeak.

Noirspeak is a companion web app for the Noirscape universe but it is also a fun tool for anyone interested in the subject matter. I intend to continue development of the tool so that people can easily contribute to it, share on social media and so on.

The name Noirspeak is inspired by other fictional registers or languages along the lines of Newspeak, from George Orwell’s 1984 or Nadsat, from Anthony Burgess’s A Clockwork Orange.

https://noirspeak.com webpage screenshot
https://noirspeak.com

References & Resources

Appendix 1 The Mition Picture Production Code (as published 31 March, 1930)

(accessed April 2021)

http://iantregillis.com/words/a-hardboiled-slang-glossary-for-something-more-than-night/greetings

(accessed April 2021)

Kevin Drum, (2011) Oh, That Old-Timey Movie Accent!

(accessed April 2021) Robin Queen, 2015 Vox Popular, The Surprising Life of Language in the Media

AR Gamification : snaps

Finding hidden objects in an augmented reality [AR] experience is fun; especially when the items have interactive qualities which trigger new content, clues and narrrative. Furthermore, given the quality of 3D assets, AR lighting, shadow and reflection prediction and generation; coupled with a the Noirscape black & white filter, it’s a great little feature to be able to capture an augmented view within the home and share with others on social media or messaging.

The user may take a picture of their AR Scene and/or re position the object to their liking
The user then reframes the image to their liking.

The Noirscape snap can be shared across social media or installed apps using the device’s native Share API
The image is now used within the app’s inventory for current user and may be shared to social media.

The functionality allow the participant to to take an AR photo snap of their new found fictional objects withing the realworld space of their home. The image can then be shared on social media or sent to a recipient using the device’s sharing API. This provides content for the user to share, a memory from the experience and marketing value for the app brand.

Noirscape AR shot shared to Twitter from within the App

In the above image, the dots which represent the AR Plane object (the horizontal plane detected in the real environment). Using the the Flutter ArCore package, there was no way to hide these once the AR object had been placed and it spoils the photograph somewhat. Fortunately I was able to fork the main package and add the functionality and send over a pull release request to the main package maintainer. So, I am now able to hide these dots just prior to takeing the capture.

One of the advantages of Flutter is the great community and the opportunity for contributing to packages.

AR : Fiction or Virtual?

Augmented Reality (AR) provides a way to place realistic looking virtual objects into a realworld scene. While the object may merely exist upon the screen of a phone; there are features to AR which combine the worlds; fiction and reality; beyond the two dimensional surface of a smartphone. For example, when a 3D object is placed within the AR scene; it may not be physically present upon the targetted surface; but certain qualities of the said surface are projected into the machine generated final composition (horizontality, width, height and distance of the plan) and ultimately this data is processed within the mind of the person who momentarily accepts the presence of a fictional item within their immediate realworld environment.

There are a few ways to look at this. One way is the simple matter of tricking the mind. I myself, during testing of my AR functionality had a fall [no long term damage to me or phone] while walking about and testing placement of fictional virtual items in my home. My perception of the realworld space around me was tricked by the magic window of the smartphone through which I had been focusing for several minutes while placing a 1940’s vintage telephone on various surfaces. I got confused and tripped over. Maybe it was just me being clumsy; but there is, it seems to me, something to be said about the nature of fiction and how we, as humans, easily incorporate representations of real items into our mental processing of reality. This isn’t even new, or a result of the emergence of hitech. Whenever we we look at a photograph, we are looking at paper and markings; yet we see a person, or a real thing. Everyone recognises Magrite’s famous painting on this subjec; The Treachery of Images.

La Trahison des images, 1929; René Magritte

In Magritte’s case, the representative object in question is more evidentally two dimensional and contrived. Although the mind interprates the markings as being a placeholder for a real pipe; that is all it is – a placeholder. In the same way that the word P I P E is a placeholder for the physical and usable real thing. In Magritte’s painting, the image in question is a question of linguistic semantics. The children’s book style in which the image is constructed intends to poke fun at the way we learn to identify things from images in the same way we do from words; in this case the pipe image is little more than a word; a modern hieroglyphic.

About the same time as the famous illustration, Magritte published a fascinating article in the newspaper, ‘The Revolutionary Surrealist’ entitled ‘les images & les mots’ (‘Words & Images’) in which he shares a number of observations or platitudes, maybe, surrounding the nature of words, images and their role in our interpretation of reality.

Magritte,  1929, Les mots et les images, p. 32

The above illustrations are about depictions of reality and the way in which our minds relate to the concept of things; particularly within the scheme of language and words; but also just the nature of things. For example, one image remarks how an object leads one to believe that there are other objects behind it. Or, from another page, how the visible contours or objects, in reality, touch one another in a mosaique manner.

Magritte, 1929 Les Mots et les Images

So, what impact does Augmented Reality have, in respect to these kinds of platitudes? the expectance that objects hide other objects and so forth. AR techniques allow devices and software to imitate reality and then embed the imitation within reality; capturing the direction of light within the scene, to cast convincing looking shadows and reflections. In the case of Noirscape; a participant discovers a fictional telephone and can place the telephone in their realworld environment rather convincingly. The app also features a rotary dialler which is associated with the fictional telephone. Within the scope of the app; the dialler is used to call fictional characters. But, if it were to be connected to the device’s real calling capabilities; in other words, the participant can call someone in the realworld through interacting with the representative dial of the AR telephone; then its hard for me to make a distinction between using a physical phone or the augmented reality one. I think maybe, in this case, it is no longer a question of being a fictional telephone; but rather a virtual telephone. Whereas Magritte depicts a pipe, I cannot smoke a mere depiction. Whereas, I could hook up my fictional telephone to the realworld and make a call.

3d telephone from Noirscape App AR feature

References

http://ideophone.org/magritte-on-words-and-images/ [accessed 25/03/21]

https://plato.stanford.edu/entries/fiction/ [accessed 25/03/21]

http://ideophone.org/description-and-depiction/ [accessed 25/03/21]

360 Flashback Interaction

The challenge is to keep the app’s media files small so that they can be served on-demand from a content delivery network (CDN). This ensures only content that is relevant to the current user is downloaded and the main app size is kept small. Instead of using a mp4 or other type of movie file to convey the town 360 scene I am using the new Google webp format which I have previously validated within the 360 Flutter component. My aim is to keep this file as small as possible so it can be quickly served from a remote server. Other content such as voice and 3d characters which are part of the experience but not specific to a town  may be compiled as assets with the build as they will not need to update as often as other interactions. There is also the option to stream the non-town specific video over the underlying 360 animated webp image file.

Using an Adobe Media Encoder plugin I was able to export a short section of my 360 movie into a webm file. The movie version of the webp format. However, this format turned out not to be supported by the 360 component I am using. So, I am looking to convert to webp as I will not need embedded audio; which can be played from a separate file.

I found that Google provides decent documentation for webp as well as a number of command line programs to help convert.

https://developers.google.com/speed/webp/docs/using

The conversion worked nicely. I am now able to open up my town scene with movement; in this case rainfall. However, (there is always a however…) the 360 plugin supports the standard Flutter Image widget which in turn support webp animated images, but I am so far unable to loop. So, the animation stops after the final frame.

The tools I am using can be downloaded here:

https://storage.googleapis.com/downloads.webmproject.org/releases/webp/index.html

Instruction for configuring the lightweight image sequence with Google tools followed by deployment to the CDN

Adding Sound and Interaction

As the 360 visual content of the spatial narrative is town based; the relevant media file is served from the CDN. Its important that the image files are as optimized as possible do they transfer quickly. Sound effects will be embedded within the app as they are common to all users, whereas narrative speech is also, like the town based visual content, based on the user’s language but also the path they take through the adventure based on their decision making. So, it also makes sense to serve this content from CDN. Speech files do not need to load instantly as they are usually triggered through an interaction. So, the final recipe involves playing embedded sound effects and an embedded visual effect while the main 360 media loads. The media is then cached so the slight delay is only noticable on the first playback. The narrative sound is then played on top of the 360 visual content at the appropriate moments.

For the sound I have opted for an AI generated voice; albeit with intonation and a Hollywood accent.

Interactive 360 Scene Demo

Spatial Narrative Content

Noirscape experiments with a cross reality approach to narrative. Participants search for and find fictional objects in their own physical home through the AR (augmented reality) feature. One of these objects is a door and it’s keyhole a doorway between an augmented view of one world and the entirely fictional world of another. Furthermore, Noirscape binds the narrative with physical spaces in the nearby town. In my case and for the purpose of the pilot version of Noirscape, this is the French town of Bourg-en-Bresse.

I previously carried out fieldwork collecting 360 panoramic photographic content in and around the town; at over twenty locations selected not neccesarily for their prominence, but also for their intrigue; whether that be a curious street name, a bygone brewery turned warehouse or the site of a house where a celebrated artist and philosopher one lived. The noirscape experience will take participants into their town where segments of narrative are reveals through what I call flashbacks – these are a staple component of film noir; where the protagonist, who is usually since deceased or condemmed to a life of prison, recounts events from the past which played an important role in their current whereabouts [or absence of].

Opening Sequence to Sunset Boulevard, Paramount, 1950

My challenge is to take my 360 content from the town and combine it with fictional Noir narrative to give an augmented or combined immersive experience whereby the content is triggered only by visiting the place in the physical world and at which point, a flashback from a fictional past occurs. To achieve this I decided to work with digital 3d character creation and animation. I had previously arranged to work with a friend who is also an actor; but, it’s complicate right now with the pandemic, to meet up and spending enough quality time to get something filmed; I was planning to use my green screens and then take the content into the 360 editor using Adobe After Effects and Premier Pro. One thing lead to another and I opted for digital characters. I initially hoped I’d be able to use Adobe software but they have discontinued their Fuse product which was a character designer app that could be used with Mixamo, their recently acquired character animation service. I decided to use Reallusion’s Character Creator instead due to the vast amount of resources available. I used Headshot, their AI face generator to base character on my own face (although I’ve reworked it somewhat since!) and I imported custom objects like a Fedora hat and set up the character in a black coat.

A base character in Reallusion Character Creator software with an AI interpretation of my face projected onto it.
My clothed and hatted character in a T pose
Closer shot

Experimenting with different predefined pose templates

Next I took the character into iClone, Reallusion’s 3D animation suite. The challenge with iClone was to be able to bring in my 360 photo and create my own scene within the panorama. However, I ran into problems with this at first. While export to 360 panorama format is suported in iClone, I couldn’t achieve this using photography without experiencing problems with the way the image was being wrapper; due to distortion at the poles of the sphere if the Skybox object. The Skybox object in iClone and more generally in 3D design, is the imagery used to define the farthest most visible details; this would normally be the sky, hence the name; but may also be a distant mountain range. Usually this would only be thought of as a backdrop, with far more focus on the foreground and midground detail. In my case the Skybox would be represented by a complete 360 photo, in which I would place 3D assets like a person, a vehicle, etc.

Example of 0 degrees (ground) when 360 is wrapped within Photoshop;

Ground shot taken in iClone with the same 360 photo set as the Skybox image

I discussed the issue in the Reallusion support forum; and one solution put forward was to create my own 3d sphere object and set my 360 image as the texture. This did produce a slightly better outcome but not satisfactory enough for what I need. The Reallusion is fantastic nontheless; what I am seeking to do is certainly not a typical user-case by any means. One really good feature with iClone, and one of the key reasons for settings a photo as the Skybox, is for calculating light within a scene. The iClone software will identify from the image, in my case the 360 photo, which direction light is coming from and therefore where to cast light and shade within the 3D assets added to the scene. So, although I chose not to use iClone with the 360 photo visible, I still used it for the lighting work.

Scene from within iClone with my 3D character and other assets placed within my photo.

Within iClone I applied some subtle animation to my character; his tie blows in the wind and he blinks and moves a little while he waits for his rendez-vous. I applied rain effects with splashes and flickering light effects. In order to export my animation without the Skybox image so that I could bring it into Adobe After Effects I would need to export as an image sequence so ensure a transparent background. The sequence is 30 seconds long and 30 frames per second; so the software rendered 900 images in total which I then imported into After Effects.

Within After Effects the first challenge was to align the two-dimensional representation of my sequence within a 360 environment. If I place it as-is then it will be foreably bent into a banana shape as it is interprated through a 360 viewer. So, to avoid this, it’s important to alter the curvature of the 2d assets to align with the 360 image in equirectangular-panoramic format.

The 2D animation curvature is altered to match that of the 360 scene so that when wrapped into a sphere it looks correct.
My Animation positioned within the 360 photo with field of view warping to match 360 sphere position.
Adobe After Effects Settings Using the VR Plane to Sphere effect to warp the field of view.

I’m generally pleases with the outcome and although it took quite a bit of time to get what I wanted, I now have a documented workflow for the process; I have a character ready to deplot to new scenarios and the knowhow to create others much more quickly. A small issue I have with the end result is that the animation is too subtle to really see properly on a mobile device; but this is easily tweaked. For now, I’m going to settle with what I have for the purpose of integrating with the app. The next step is to create a looping image based version of the scene in webp format as I have shown in a previous post. I will then play the audio channel, with the voice narration and sound effects via the app/device rather than the media file itself. This will keep the size of the media file down and allow me to serve the localised element (the view using footage from a specific town) and the global content – the spoken narrative.

Mobile phone view of interactive scene
Interactive YoutTube Version

Target Audience Research : User Survey

While the allure of Noir is an assumption I hold about the target audience; it is just as important that the audience is interested in role-playing, interacting with other participants and within the context of their local town. I made assumptions about attitudes towards a product which is tailored to a person’s home town as well as the concept of leaving one’s mark withing the fictional world of the experience. In order to test and respond further to these assumptions I carried out some primary research which I have detailed below.

Although I only polled one hundred people; I used an initial qualifier filter to exclude people who answered no to the question “Could you be interested in a themed digital interactive experience?”.  83% of the initially filtered participants said they would participate in the digital interactive experience if it was to be Film Noir themed. I then tested the perceptions, knowledge and preferences that the poll participants held about both Film Noir and the nature of the type of app I am building. Among the finding were that a majority (61%) would prefer to download the app via an app store and print out any physical assets rather than purchase a boxed product. This is not to say that a boxed product is unpopular; 39% would prefer a boxed edition. However, the survey did not give details of pricing differences. While my original assumption early on in the product development cycle has been to ship the app as a hybrid  boxed app. I have come up against logistical and cost problems with a supplier as well.  Therefore it is very likely, with the data from the research, that the app will initially be distributed primarily via the app stores.

Demograhics Summary

Several other assumptions were validated within the research. 59% of participants prefer an open narrative as opposed to a prewritten one. Although a greater majority (77%) prefer a narrative with an ending. This latter finding suggests that an episodic approach could satisfy a large majority; whereby the participants have control over much of the narrative experience while encountering closure through periodic narrative endings to sub-plots. A staggering 82% of those surveyed agreed to some degree that it is better when a digital interactive experience is tailored to a participant’s local town or city. 62% of participants said they would be either moderately or extremely likely to engage with interactive digital content within their local town as part of the experience; a further 22% said they would slightly likely. Leaving one’s mark within an interactive experience, such as by leaving hidden messages at locations for other participants to discover, was deemed between moderately and extremely important to 82% of people. Those surveyed agreed overwhelmingly (87%) that digital interactive experiences through mobile devices could encourage people to engage with their local towns more if tailored to included familiar content such as well know local buildings and places.

Using the demographical data obtained from the survey I was able to determine certain trends which helps to better identify more specific target audience cross sections.

92% 18 – 24
81% 25 -34
84% 35 – 44
69% 45 – 54
71% >54

While the idea of a digital interactive Film Noir experience is popular in all age ranges; it is most popular with younger participants.When asked about the likelihood of engaging with digital content in their local town; the results where consistent among age groups.

67%  18 – 24
53%  25 – 34
68%  35 – 44
67%  45 – 54

Generally, across the results there was a suggestion of some scepticism in the 25 – 34 age group and the greatest level of interest in the 18 – 24 while over 35s were generally in between these other two groups in terms of levels of interest.There was no disparity in the results in terms of whether the person was using iPhone or Android. Some other demographic filters produced some variations when answering yes to the question whether they would want to participate in a Film Noir experience.

93% Uni
93% PostGrad
76% High School
81% Single
96% Married
96% Male
80% Female

Overall the ideal target candiate, based on this data would be:

Male, Married, University educated and over 35.

Although all demographics in the 18 – 24 age groups responded positively across the survey, the quantity of participants polled is limited; but it serves as a litmus test for the general take up of the idea. Finally, there was one further result which I found to be noteworthy. I asked participants which other themes would be of interest to them other than Noir and the most popular response out of ten choices was Comedy (64%) and Romantic (43%). These are elements which could and perhaps should be included within the ongoing development of the gameplay of Noirscape. Noir film was certainly not without a sense of humour, albeit a dark one.

I always cry at weddings. Especially my own.

Humphrey Bogart, Film Noir Actor

Raw Data: Survey Results from March 2021

The first question seeks to get an idea about the general public’s perception of what people think Film Noir is. The keywords
‘love’, ‘detective’ and ‘crime’ came out on top along with ‘black & white’. This was more of an intro question to serve as a guide and a reminder as to what
is meant by Noir for the survey the participant is about to engage with.

I was interested to understand about perception of Noir in terms of suitability for younger audiences. My assumption is that the app would be for over sixteens.
But, it’s also true that at the time of the height of Noir cinema there was tight rules and regulations regarding content and language. Writers and directors made creative use of language and lighting, scenes and so on to evoke the ‘forbidden’ aspects of the content, without breaking the rules.

I wanted to learn about what an audience would do when given the power to drive the narrative themselves; to understand which Noiresque scenarios could be most popular.

The 100 participants who took part were pre-filtered. They all had previously agreed that they would be open to participating in a digitial interactive experience.
However, they were not told of the theme. So, question four is a good test to see whether the Noir theme is alluring or not. Those who answered no, were then taken straight to question 6.

Question 5 expands further on seekings to understand audience perception and preferences. This time in relation to role playing.

Question 6 was answered exclusively by those who said they would not wish to participate in a Noir themed digital interactive experience.
Six of the initially pre-filtered 100 participants seemingly changed their minds about openess to such experiences. Just 9 out of the initial 100 stated they didn’t like the theme.

Participants who stated that they didn’t like digital interactive experiences in question 6 were not asked any further questions. The remaining
94 participants were then asked about other themes they would be interested in either as well as Noir, or in the case of the 9 people who don’t like Noir, instead of it.
This question provides a good indicator of where Noir theme stands in relation to other more or less [assumed] popular themes.

The remaining questions were theme neutral and sought to find out more about what the audience’s perception of the nature of digital interactive experiences;
and some specific questions about assumptions carried by my proposed app.

Interactivity between participants would be a popular feature. 73% would prefer this. But, 27% is a significant enough proportion to warrant further analysis.
What are the reasons why, for example? Can those reasons be overcome by reassurances? Should the app offer a solo mode? This piece of research does not answer those questions.

The following question is troublesome is some ways, to the premise and assumption that the Noirscope experience can be ongoing and expanding.
Interestingly, it’s a very similar distribution to the previous question and further analysis could be done to look at any correlation between the datas from these two questions.

Question 12 validates an important assumption and premise of Noirscape – which of course is designed to be tailored to a participants locality.

Question 13 re-enforces my locality hypothesis further [along with Q 12].

Leaving one’s mark in a digital interactive experience is import to a large majority of people questioned.

Finally, along with the Q12 and Q13 the user response confirmed the assumption that an experience with elements tailored to ther local town could encourage people to engage more with real local places.

Interactive Rotary Phone Demo

I added some final sparkle to my vintage rotary dialler. Using various calculations I am able to predict which number has been dialled in a way this quite closely emulates the real thing. For example, a digit is only registered when the dial is turned fully to the catch.

I also added sound effects. Each digit has its own sound file to correspond with the length of time of the rotation which of course varies for each number. The sound effect also has to respond to when the dial is released too early and therefore has an irregular return rotation timespan.

Lastly, I added some texture to the background and some text using an style typewriter font which shows the participants name and their own tel number within the experience.

I had great fun putting this into the hands of my two childre, aged 9 and 12; they had no idea how to work a rotary dial phone. Who needs to invent enigmas and puzzles when we can just emulate the trials and tribulations of analog technology!

I also shared a demo with friends and colleagues as well as on social media and I was quite surprised by the number of people suggesting this could become an app in it’s own right. I haven’t researched what’s already in the app stores. But, certainly an interesting idea would be to make a standalone rotary dialler app which interacts with the phone’s call api to initiate real calls. I am already thinking about the fun that could be had creating different version; the 1970’s, 1890’s, 1950’s.

Anyway, here is a video I put together to demonstrate the feature in action.

https://www.youtube.com/watch?v=Kr54QQu3U4Q