Immersive and Sensory Media
Heidi Boisvert – The Socio-Cultural and Neurobiological Impact of Emerging Forms of Storytelling
Daniel Fetzner – Improvising in/with/through 360° Video
Craig Hight – Tools for augmenting reality
The Socio-Cultural and Neurobiological Impacts of Emerging Forms of Storytelling
Amidst the flurry of hype surrounding VR/AR/AI/ML in documentary film, games for social change, journalism and clinical research, little attention has been paid to either the long-term socio-cultural and neurobiological impacts of the technology. Proponents across diverse sectors believe these latest emerging forms of storytelling can serve as an “empathy machine” capable of radically transforming hearts and minds. From reducing implicit bias to addressing serious psychological and sensorimotor disorders, many over-arching assumptions have been forwarded about these emerging forms without conducting rigorous scientific research or engaging in a critical discourse about possible ethical implications of presence, such as vicarious trauma or poverty porn. This evidence-based talk will, therefore, share the development and findings of the Limbic Lab, which employs integrated biometrics with story coding and predictive modeling to isolate the common variables that move us to act by identifying the cognitive and affective cues which enable or disable empathic engagement and prosocial behavior.
IMPROVISING in/with/through 360° Video
»The alternative to totalisation is not fragmentation, rupture and discontinuity. It is rather a holism that is
anti-compositional, fluid, processual and improvisatory.« Tim Ingold, Being Alive p. 226
We consider the handling of 360° video as an improvisational act. The technology is
mediating the processual logic of the computational with the sensomotoric scheme of the user
wearing a HMD. By improvising with a 360° video sequence, interruptions, interferences and
intervals are the aesthetic magnitudes of an unpredictable way finding with an open ending.
What might occur as a contradiction is in fact the core point of any improvisational act – in
music, dance or thinking. By its entanglement with the motor scheme this interaction comes
closer to the metaphor of surfing as a kinesthetic experience than any desktop application.
In this sense navigating through a 360° video in generating a senso-motoric meshwork with
the user that is generating a flow of material substances that are not just topologically fluid but
also a-narrative. Deterministic structures and coincidences complement each other as they do
in the tradition of direct cinema.
We understand this approach as a proprioceptive symbiogenesis between application and
user, formerly described as »living documentary« (Gaudenzi 2013) and »programmatic
choreography« (Miles 2017). In our experimental setup SNOW WHITE we focus on body
experiences with the focus on environmental relations. This serves as a preliminary study for
the artistic research DARK MIRROR which is planned in 2018/19.
Tools for augmenting reality
Apple Corp’s support for AR development, through recent releases of their iPhone and iOS
ecosystem, offer further evidence of a gradual drift in AR capabilities toward the seamless
integration of graphic elements into screen-based ‘reality’. Rather than a dependence on
heads-up displays that overlay familiar forms of information design, more recent iterations
of AR offer the promise of designing digital artefacts which appear as 3D objects which ‘sit’
within the users’ immediate social-political environment when viewed through a screen.
These kinds of hybridised visual approaches are already familiar within documentary
practice, particularly from the long histories of documation, drama-documentary,
mockumentary, docugames and wider experimentation with technologies such as CGI. Their
play with issues of indexicality, and strategies of reconstruction, simulation and the
warranting of social-historical knowledge have long been part of an expanded realm of
media production techniques available to nonfiction content producers.
Augmented Reality (AR) similarly offers the potential to extend, translate and transform key
aspects of documentary practice, but in this case within the bounds of possibilities
generated in collaboration with software-based tools. As I have argued elsewhere (Hight
2017), a variety of software-based forms of interactivity have emerged which need to be
considered as human-software assemblages operating within the context of other
developments within software culture. AR is one of a series of possibilities for users to
‘interact’ with and through digital platforms, performing types of agency which are
generated in the first instance through designers’ collaborative practice with available
software development tools. Designers co-create with specific sets of software-based
toolsets, with their own conceptual frameworks and hierarchy of affordances which
intimately inform the possibilities for the resulting AR experiences. This presentation will
explore the implications of Apple’s ARKit (https://developer.apple.com/arkit/ ) as a
software-based platform for designing augmented realities.
Andrejevic, Mark, and Mark Burdon (2014) ‘Defining the sensor society’, Television & New
Media Vol. 16(1): 19–36.
Hight, C. (2017) ‘Software as co-creator in interactive documentary’, in Judith Ashton,
Sandra Gaudenzi and Mandy Rose (eds.) the Evolving Practices of Interactive
Documentary, Wallflower Press, pp. 82-96.
Hoelzl, Ingrid, & Rémi Marie. (2015) Softimage: Towards a New Theory of the Digital Image.
Intellect Books.
Lehmuskallio, Asko (2016) ‘The camera as a sensor: The visualization of everyday digital
photography as simulative, heuristic and layered pictures’, E. G. Cruz, & A. Lehmuskallio,
(Eds.) (2016) Digital photography and everyday life: empirical studies on material visual
practices, London, Routledge, pp. 243-66.
Manovich, Lev (2013) Software Takes Command, International Texts in Critical Media
Aesthetics, Vol 5, New York, Bloomsbury Press.