At Knowle West Media Centre (KWMC), we’re always exploring how technology can be shaped with – not just for – the people who use it. So we were keen to get involved when the opportunity arose to support Cognition, a Bristol Digital Futures Institute Seedcorn project led by artist-researchers Dr Lisa May Thomas and Eirini Lampiri.
Cognition asks key questions about accessibility and inclusion in virtual and mixed reality experiences. It challenges the assumption that this tech must be designed for people with “normative” bodies or senses. Instead, it explores how these experiences might be collaboratively reimagined with people who are often excluded from design processes.
KWMC helped connect the project team with local participants and provided space and support for their workshops.
Find out more in Lisa & Eirini’s blog below. We’re excited to see what happens next!
BDFI 2024 Seedcorn Project: Cognition – Making Mixed Reality Experiences Accessible
By Lisa Thomas and Eirini Lampiri
What is the project about? What issues/ themes does it address?
Virtual Reality (VR) technology offers the potential to be a transformative tool. We ask, what are the current issues in ethically, responsibly and inclusively shaping the design and use of this technology in its increasing take up in society?
VR manifests notions of embodiment and immersion but impedes/limits modes of sensing and only engages with specific body parts to interact with the (simulated) world, communities and audiences that are represented or engaged with are often excluded from design processes.
1. VR is designed for people with normative sensory systems/standpoints, marginalising certain people. The design and use of VR rests on normative assumptions that prioritise sight over other forms of bodily sensing. VR’s promise of transportation to other realities seems to mainly depend on the participant’s eyes, requiring a sensory system that is visually dominant.
2. VR limits people’s experiences of their bodies and neglects the importance of the body in how we experience the potentially behaviour-shifting simulated virtual worlds we inhabit from inside a VR headset. Current VR systems assume the user to have a non-disabled body and is consequently inaccessible for many people. Whilst there is a willingness to accommodate different bodies in VR, the fundamental approach/system does not change.
3. Designing VR systems and worlds often takes place without its intended audience-participants/ outside of the communities it is representing. Excluding people from the experience design process results in poor design and user experience (UX). Whilst VR experience makers are now considering co-design processes, we ask: who gets to participate in these processes?
What did you set out to do?
With Bristol Digital Future Institute (BDFI) 2023 Seedcorn funding, Dr Lisa May Thomas and Eirini Lampiri undertook practice-led research to explore how VR might be more accessible for potentially excluded people, specifically developing and co-designing VR-practices with visually and kinetically impaired participants. They used existing Mixed Reality (MR) prototype, Cognition, created by Eirini and creative technologist, Julia Ronneberger alongside using existing practices developed by Lisa May Thomas as part of her social VR experience SOMA.
Key areas explored through this process were:
- notions of embodiment and immersion in VR, what these terms mean for different people
- multi-sensory on-boarding and off-boarding partner practices
- how we talk about/describe our sensory experiences
- incorporating haptic and bodily interactions within a VR experience
- investigating the relationship between physical and virtual worlds/environments and bodies
- exploring and discussing the effects of sensory alignment and misalignment in VR (and also in everyday lives) using the mapped/aligned physical and virtual worlds of Cognition as a starting point
- understanding what it is like to share our bio-data with each other using Empatica wristband sensors
What is Cognition and what technologies are involved?
Cognition combines VR narrative with a physical kinetic set to narrate a story of inequity, body ownership and free will. To enable participants to interact with the narrative using their hands, and with minimum technical barriers, Cognition employs an unusual combination of technologies; this includes the Vive VR headset and Vive trackers for movement and rotation tracking, the Leap Motion controller for hand tracking, potentiometers and Arduino for tracking the door movements, and 3D printing creating precise physical sets.
How did you do it?
1. interdisciplinary team
We collaborated with artist and programmer, Joseph Wilk, whose practice interrogates accessibility and inclusivity aspects in sociopolitical and technological context, MA Immersive Arts graduand, Shivani Despande and creative technologist Julia Ronneberger.
2. R+D workshops
We invited colleagues and experts from across social sciences, engineering, and arts disciplines across academia and industry to explore some of our early ideas for how we might approach designing sessions for the upcoming community-based workshops with visually and kinetically impaired people at KWMC.
3. Community workshops
We started the community-based workshops at KWMC by trying to explore and understand the role of technology in our visually and kinetically impaired participants’ lives. We were surprised to realise how much integrated technology is in their daily reality – from smart wearables to talking appliances, and we discussed what VR is, how it works, and ultimately question why visually impaired people would want to experience/ use VR.
Our technological discussions became hands-on explorations by passing along to the participants all the technical equipment that fuels Cognition, facilitating participants to familiarise themselves with the VR kit as part of our on-boarding process. The on-boarding continued with participants haptically exploring the physical set before inviting them to explore a variation of complementary roles, experiencing the physical and digital world of Cognition inside and outside of VR.
During the first workshop, we opted to initiate participants in the Cognition world without sharing any of the experience’s narrative information and backstory. This choice was designed to inspire participants to prioritize the sense of touch and use it to create their own interpretation of the Cognition world, based on the physical set that they could haptically explore, and the digital world seen in the VR.
Big part of the second workshop was dedicated to reflecting on the participants’ experience from the first workshop, discussing our choice of not sharing any narrative information about Cognition before participants entered its world. Participants found it difficult to connect with the Cognition world without any verbal information. They found challenging our invitation to prioritise touch and they expressed their preference for audio description.
Integrating the participants’ feedback, the rest of the second workshop focused on giving participants more time to explore the physical model of Cognition in an analogue way, prioritizing touch but also delving into describing the physical and digital world of Cognition. Participants asked questions about the backstory of Cognition and they were invited to create clay characters and elements that they wish to integrate into the physical set. We then welcomed participants to use their every-day technology to capture a photo of Cognition with audio description and explore it through the lenses of a smart camera as well.
Workshop three focused on delving into worldbuilding and the backstory of Cognition’s world as an experience. Participants were curious about the cruel conditions in the world of Cognition, found personal connections to the story and envisioned new and more hopeful narratives for this world and its Characters, using the clay elements they previously created, to save the characters and present them with alternative options and escaping opportunities. By the end of the workshop, participants were excited about the overall experience and eager to keep being part of any future developments of Cognition as an experience.
What did you find out?
Through a valuable engagement with the people who attended the r+d workshops, we learned that the sequencing of on-boarding activities is important and highly influential to a VR experience, and this requires in-depth planning and testing in setting up an engagement with VR, particularly thinking about people who have not experienced this technology before and also people with access needs.
We noted that everyone has different on-boarding needs, and that practices which enhance/ tune sensory attention/ awareness require time. Even though we prioritised non-visual senses in our activities, we found that vision was still the dominant sense and that visual cues were important to people to enhance a sense of connection.
Whilst we did not centre the story of Cognition, it became an important factor for people in the workshops, helping them to connect to what they were doing and eliciting further conversations and questions about the use and effectiveness of this technological medium to tell stories.
The notion of sensory misalignment was discussed bringing up lots of different ideas and meanings for people in the room, from considering the misalignments that happen in VR (between physical and virtual worlds), and also outside of VR, in daily lives.
Lastly, and most importantly we talked in depth about how we would engage meaningfully with the group at KWMC and the importance of coming together with them to co-create a code of conduct – to agree on a way of being together that foregrounded trust, openness, and clear communication as a starting point for these sessions.
With the KWMC participants, in setting up our code of conduct with the group, we discussed the importance of clear communication, setting out expectations, and not being afraid to ask for more information, clarity and guidance. To not make assumptions (about needs and capabilities), and to be open to alternative viewpoints. We talked about how we might together find ways to share ideas, be creative and open to change. We noted that it was all older male participants that came to the sessions, and that technologies were already prevalent and pervasive in their lives. We talked about their use of audio description (AD) in their everyday lives through phone apps (such as Be My Eyes) and also, for example, when they go to the theatre or see a film. When this works well for them and when it doesn’t, and why.
One of the key learnings we took from working with the participants at KWMC, was how to find the balance between giving them (audio descriptive) information about the visuality of the Cognition world and leaving room for their imaginations to create different worlds through their non-visual senses, – predominantly sound and touch i.e. hearing and touching the physical kinetic set of Cognition. Whilst their need to access the visual world was clear, and initially, they were questioning and finding it challenging to access the world of Cognition without the reliance on AD. After a while though, and through touching the physical set of Cognition and exploring the visual and narrative realms of their own imaginations, they dreamed up and storied new worlds, finding positive resolutions, accessible, caring choices in response to the difficult narrative ending question posed to audiences in Cognition:
In a world made of flesh and gears that demands more than you have to give, what are you willing to sacrifice?
Thanks to
BDFI and to our project partners:
Pervasive Media Studio, Knowle West Media Centre, Ultraleap, MyWorld
Our collaborators:
Joseph Wilk – Creative Technologist
Julia Ronneberger – Creative Technologist Consultant
Shivani Deshpande – Junior Creative Technologist
And all of our participants