Whole Body Interaction at the V&A

The V&A Museum has an exciting exhibit currently open in the Sackler Centre. ‘Digital Dragons’ makes two Chinese paintings ‘come alive’ through digital projection and a whole body interface supported by Kinect devices. Stepping on the floor influences the projection of the painting: depending on where you step, the landscape painting gains colour and human activity and in the other painting, dragons begin to chase coloured pearls.

At MODE we’ve been observing children as they interact with the exhibit. In a series of naturalistic observations, we videoed 1-4 year olds interacting with the exhibit with their parents. In the next phase of research, we observed 7-8 year olds from the same school interacting with the exhibit, and then interviewed them afterwards to see how they had made sense of the experience.

Our questions in conducting these observations have been broad. We’re interested in how children make sense of their movements in relation to the projected visual activity; how the involvement of the whole body in this experience might influence the children’s interpretations of the paintings involved in the exhibit; and how social interactions are configured and coordinated in this type of interactive museum space.

In-depth analysis will begin soon. Our preliminary thoughts and ideas have centred on where the students focused their attention. For the first cycle of the interaction with the paintings, many of the children seemed to focus exclusively on the colour and movement they saw on the floor. Over the course of their interaction however, they became increasingly aware that their movements were having an impact on the projected image and began to make sense of exactly what this relationship was. This might suggest that exhibits that make use of two separately located sites of activity (here the floor and the wall) may require sustained interaction if students are to understand how activity in these spaces is connected.

What is Somaesthetics?

I found jotting down 10 things I know about phenomenology a helpful exercise for consolidating and summarizing knowledge of a central approach in embodiment research. Over the last few weeks, I have repeatedly come across the term ‘Somaesthetics’ in relation to embodiment and have done some preliminary reading on the term. In an effort to make sense of the fledgling understanding I’ve gained, I decided to try the ’10 things I know’ approach again. Without looking at books, notes or papers, I scribbled down the following:

1. Somaesthetics is an emerging discipline primarily proposed by Richard Shusterman, who argues for it to sit within Philosophy, and more specifically the philosophical sub-discipline of Aesthetics.

2. Somaesthetics as a discipline foregrounds the role of bodily experience in aesthetic appreciation. Shusterman argues that it will offer a more substantial and systematic framework for research on the body.

3. Somaesthetics comprises three branches: Analytic somaesthetics; Pragmatic somaesthetics; Practical somaesthetics

4. Analytic somaesthetics comprises interpretive studies of bodily practice. These occur on a theoretical level and would include research linking the body and bodily practices with the organization of society and the individual psyche. Shusterman argues that the work of Foucault, Bourdieu and Merleau-Ponty would all constitute analytic somaesthetics.

5. Pragmatic somaesthetics comprises bodies of thought that suggest ways of training or harnessing bodily experience. Practices such as Tai Chi and the Alexander Technique are accompanied by ideas and theories of the body and these would belong to this branch of pragmatic somaesthetics.

6. Practical somaesthetics consists of the practice itself – the Tai Chi, or the yoga, or the Alexander Technique. It is not entirely clear which bodily practices Shusterman chooses to accept within practical somaesthetics. He seems to place less value on athletics or football, and much more on solitary practices that reference bodily understanding more explicitly.

7. In foregrounding the body in aesthetic experience, Shusterman argues that a distinction needs to be drawn between representational foregrounding and experiential foregrounding. In the former, the body is seen and treated as an external object. In the latter, the body is a fundamental part or vehicle of lived experience.

8. Critiques of somaesthetics include the linking of the discipline to normative approaches to the body. Somaesthetics places a value on the physical cultivation of the body and this is reminiscent of doctrines that treat the body as an object to be molded into a form that conforms to the social ideal.

9. Shusterman’s response to this critique rests on the distinction he makes between representational and experiential foregrounding of the body. Thus, he argues that cultivating the body as it is experienced (rather than as it is represented) does not relate to social norms, but rather to the improvement of individual perception and action.

10. Shusterman advocates that there is a role for somaesthetics in education since it enables us to ‘feel better’ in both senses of the phrase. Cultivating bodily practices heightens our sensations and perceptions of the world around us; they also encourage us to be more in control and more caring of the self.

The combination of theoretical and practical approaches to the body certainly makes the discipline of somaesthetics an interesting prospect, but I am left with questions about the possibility and worth of trying to merge these perspectives. Shusterman’s framework makes such a sharp and convincing distinction between analytic, pragmatic and practical somaesthetics, that it isn’t clear how it draws these together at all. How are Foucault’s theories of the body related to the theory or practice of yoga? How can somaesthetics help us to access this relationship? Is this a relationship that should exist given how different the objectives related to each project are? Certainly, I believe that theorists of the body can learn much from bodily practices and practitioners who work with the body, but what is the benefit of binding them together in a single discipline?

Shusterman, R. (2008) Body consciousness: A philosophy of mindfulness and somaesthetics. Cambridge: Cambridge University Press.

Shusterman, R. (2006) Thinking through the body, educating for the humanities: A plea for somaesthetics. The journal of aesthetic education, 40(1), 1-21.

Making Strange: Experiments in Bodily Awareness

By Mona Sakr

An overview Loke and Robertson’s ‘Making Strange’ methodology and a discussion of its relevance to embodiment

In Loke and Robertson’s (2008) ‘making strange’ methodology, the way we perceive and feel the moving body is actively unsettled in order to find new perspectives on the body and bodily interaction. The practice of ‘making strange’ builds on the work of dancer Maxine Sheets-Johnstone who explored the phenomenological consequences when individuals disrupted their familiar or habitual movements. In ‘making strange’, our assumptions and habitual understandings about the body are unsettled through explicit inquiries into movement. Loke and Robertson argue that de-familiarising the movements of the body can open up new spaces in the design of artefacts and technologies. It can help us to invent or devise new types of movement; it enables an investigation into the experience of movement; and it encourages us to find new methods for re-enacting or recording movement.

 So how does ‘making strange’ work? Loke and Robertson suggest a range of techniques that can facilitate innovative thinking about the body and movement:

• Scoring – a simple score might consist of three movements e.g. walking, standing and crouching, but by improvising in terms of the speed, duration, order, focus, or use of space, these movements can be radically altered.
• Falling – unorthodox movements, like falling, can help us to re-feel and re-negotiate our relationship with the moving body.
• Somatics – certain practices, like Qi Gong, can help us to become more aware of our felt sensations and to prioritise these over the external perception of movement.
• Finding Pathways – what part of the body do you lead with when you move, and what would happen if this were changed? Try walking left shoulder-first, or leading with your elbow when you lift your arm.
• Imagery – the language we use to talk about the body and movement can help us to enact movements differently. Think of the ‘light as a feather, stiff as a board’ game that you may have played when you were younger. Chanting these words as a group while lifting one of your peers changes the experience.

For dancers, these techniques are a way of accessing, inventing and designing new forms of movement. For researchers of the body though, we also need methods for capturing responses to these movement techniques. And as designers, we need to link these responses to, or ‘write’ them into, systems that depend on movement.

Some ideas for recording responses might include:

• Video to capture the movement
• Transcriptions of the movement through visual plots e.g. plotting the motion of particular points on the body
• Photography to capture the moments that define the essential points of the movement e.g. when we raise our hands, the essence of the movement is in the fingers raised towards the sky
• In situ comments on the phenomenological consequences of the movement
• Oral or written reflective commentaries

‘Making strange’ is an exciting starting point in the design of embodied interaction. Beyond the techniques it suggests though, we need robust methods for capturing the outcomes of practising these techniques, along with guidelines for implementing them in the design of systems that depend on whole-body interaction. For me, the next step in understanding the ‘making strange’ methodology will be to try the techniques suggested by Loke and Robertson and to capture my responses in the ways I’ve suggested.  Will the outcomes be useful in thinking about and designing for whole-body interaction?

Loke, L. & Robertson, T. (2008) Inventing and devising movement in the design of movement-based interactive systems. OZCHI’08 (81-88).

Gaze as a thinking tool

By Mona Sakr

Some thoughts about the role of gaze in students’ historical inquiry a digitally augmented local site of interest

I’m currently analyzing data from a project looking at students’ exploration of a digitally augmented local site of interest. The students used iPads to engage with the WW2 history of the local common and the experiences of people in the area during the war. One of our research focuses is how being in situ can facilitate students’ inquiry about the past. How do students use both the physical and the digital environments to support their learning and interactions?

In coming to a multimodal analysis of students’ embodied experiences of time and place, I suspected that movement would be important to students’ inquiry. I had hypothesized that movement would enable them to make links between different areas on the common. However, what I hadn’t  previously thought much about was the way that gaze would also be an essential tool in enabling inquiry about the past.

Research on gaze has tended to position it as either an indicator of attention (as in psychological research) or as a key instrument in social interaction (as in sociology and conversation analysis). But in our project on students’ inquiry on the Common, gaze acted as a thinking tool. In particular, the movement of their gaze back and forth between the digital environment of the iPad and the physical environment of the local common enabled them to engage simultaneously with the past and the present, comparing these points in time.

In an illustrative clip (picture above), two students are discussing how they think they would have felt if they had had to live in a deep shelter under the common during WW2. They talk about what they would have missed and constantly their gaze moves between the image of the shelter on the iPad screen and the physical environment that surrounds them – an environment that they describe as ‘free’. Gaze enables them to regularly re-engage with the present day environment so that they can work through abstract ideas or associations they have about the space.

To see more about this project, watch our video about embodied experiences of Clapham Common and students’ historical inquiry.

Body-thing dialogue

By Mona Sakr

Exploring the usefulness of the ‘body-thing dialogue’ metaphor for understanding embodied interaction in digital environments

In 2006, Larssen, Robertson & Edwards presented the paper ‘How it feels, not just how it looks: When bodies interact with technology’ at the Australian Computer-Human Interaction conference (OZCHI). In this paper, they suggest that  the embodied nature of interactions with technology can be accessed by thinking about each interaction as a body-thing dialogue. The body-thing dialogue is the bodily interaction that occurs between an individual and an artifact. The body-thing dialogue happens through the mode of movement and makes possible the potential for action that forms the basis of the interaction. Through this metaphor, Larssen et al. hoped to shift the focus towards the bodily nature of interactions with technologies – the extent to which these experiences are physically felt.

I would argue that the body-thing dialogue is a useful metaphor in some ways and a misleading one in other ways.

It is useful because it draws attention to the distinct and temporal nature of each embodied interaction:

• Every dialogue we engage in is different. Similarly, in human-computer interaction each interaction with technology unfolds in a specific way and in a particular context. Flow diagrams based on a non-existent ‘typical’ user do not help us to access the nature of embodied interaction.
• Every dialogue unfolds over time and can radically change from moment to moment. Similarly, each embodied interaction takes place over time and is historical – each aspect of the interaction happens in relation to the aspects that have preceded it.

But the metaphor of the body-thing dialogue is also a misleading way to think about embodied interactions in digital environments:

• Movement is a different mode to speech, with different opportunities and constraints. Is it right to apply the notion of ‘dialogue’ in the context of movement?
• Can we apply the term ‘dialogue’ to make sense of the way movements unfold between ‘a body’ and ‘a thing’? Certainly, the movements of the body and the movements of an object are not equivalents in the way that the speech of two human participants is.
• Larssen et al. suggest that the body-thing dialogue is a useful way of looking at ‘how we use our proprioceptive sense and motor skills when incorporating a tool in our bodily space so that it becomes an extension of our bodies’ (p. 2). The notion of dialogue takes us away, however, from the concept of incorporation. In a dialogue, there is self and other – the object responds to us, rather than becoming an extension of us.

So what alternative metaphors or conceptual tools enable us to think about embodied interaction in digital environments? I have yet to come across a theory that helps to frame interactions with artefacts so that the focus is on the body and felt experience, but does not trip into the pitfalls outlined above. We need to conceptualise interactions as physical couplings without using metaphors that draw on other modes of communication.

Larssen, A. T., Robertson, T., & Edwards, J. (2006, November). How it feels, not just how it looks: when bodies interact with technology. In Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (pp. 329-332). ACM.

What is affect?

An exploration of the term ‘affect’ and what it offers to embodiment research.

I am exploring the term ‘affect’ and what it might offer to my understanding of embodied experience. In this post, I consider how affect is talked about in research and think about the best way to use the concept in my own research on embodiment and digital environments.

Hudlicka (2003) presents affect as an essential aspect of embodied interaction with digital environments. Hudlicka uses the term ‘affect’ interchangeably with ‘emotion’. Thus, the need for computers to detect user affect is equivalent to their recognition of user emotion (whether the user is happy, sad, angry etc.). Similarly, Johnson and Wiles (2003) talk about the ‘positive affect’ associated with playing computer games and equate this with happiness, engagement or ‘flow’ (Csikszentmihalyi, 1992).

But Shouse (2005) argues that the terms ‘affect’, ‘emotion’ and ‘feeling’ must be understood and presented as distinct from one another. He suggests that while feeling and emotion are subject to recognition and labeling on the part of those who experience them, affect is a more simple individual response: ‘a prepersonal intensity corresponding to the passage from one experiential state of the body to another’. Affect is a one-dimensional spike in the intensity of experience.

If we use this definition, what distinguishes affect from arousal? Is Shouse simply arguing for the importance of measuring the physiological arousal of users? My understanding is that affect is more than arousal because it implies the existence of an external cause. An individual might experience an increase in arousal as a result of a fleeting image that passes through their mind, but affect is a consequence of being affected by another body, by something external. So, focusing on affect involves looking at not just the affective response but also at the stimulus that has caused it: the music that has moved us, or the image that has led to a pre-conscious experiential shift.

In the context of my research on embodiment and digital environments, I am interested in affect because pre-conscious responses to the external world are a vital part of user experience. But looking at the work of Shouse has convinced me that the pre-conscious nature of affect means that it cannot be measured by asking users to label their feelings or by recording visible emotions that I observe. On the other hand, measures of physiological arousal can offer some insight into affect but do not tell the whole story. To ‘get at’ affect requires looking at indicators of the intensity of experience (including physiological arousal, facial expression, vocalisations and other bodily cues) and linking these to the environment that surrounds the user.

References 

Csikszentmihalyi, M. (1992) Flow: The Psychology of Happiness. London: Random House.

Hudlicka, E. (2003) To feel or not to feel: The role of affect in human–computer interaction. International Journal of Human-Computer Studies, 59(1), 1-32.

Jonhson, D. M. & Wiles, J. (2003) Effective Affective User Interface Design in Games. Ergonomics, 46 (13/14), 1332 – 1345.

Shouse, E. (2005) Feeling, emotion, affect. M/C Journal, 8 (6), 26. Accessed online 24.08.2013: http://www.journal.media-culture.org.au/0512/03-shouse.php

embodied experiences of mobile technologies

An update on a current research project looking at body and space in the context of mobile technologies

By Mona Sakr

Mobile technologies have particular features that are likely to change individuals’ embodied experiences of places. Here at MODE, we’re conducting a research project to look at the influence of mobile technologies on young students’ navigation and exploration of a local site of interest and its cultural history.

We have designed a digital environment on the iPad for 10-11 year olds that allows them to explore the WW2 history of Clapham Common. The modified Evernote app, which you can see in the figure below, encourages individuals to move around the common while accessing visual, written and audio information about the history of the common. Users are also encouraged to upload their own photos and audio recordings about their experience of the common.

App environment for exploring Clapham Common

Our research questions probe the possibility that particular features of mobile technologies change the way individuals experience a place. In particular, we are interested in how these features change what the body is doing (the embodied experience of the place) and how this in turn affects the overall experience.

A mad flurry of data collection has happened over the past  fortnight. Working with 60 students at a primary school in Clapham, we collected a range of video data on their experiences of the common while using the mobile digital environment on the iPad. This data included researcher-generated video, headcam video and bodycam video. The video was supplemented with GPS trackings, and the photos and audio recordings that the students created while exploring the Common in pairs.

Researcher-generated video

Headcam video

Before and after the experience, the students recorded their thoughts and feelings about the common via a series of classroom activities, including map-making and recounting the route they had taken on a mapped floor.

Over the next few weeks, we’ll begin to get to grips with all the data we’ve collected. As well as engaging with the research questions about embodied experiences, we are trying to determine how different forms of digital data can enable us to engage with embodiment and embodied experiences of place on an empirical level.  

  

body, affect, engagement

By Mona Sakr

In a study by Bianchi-Berthouze et al. (2007), game-playing through a whole-body interface was compared with control through the hands. When participants in the study controlled the game (Guitar Hero) through their whole bodies, they reported more engagement, as well as displaying increased engagement through game-related body movements that were unnecessary for control (e.g. nodding their heads in time to the music).

Why were users more engaged when control of the game involved their whole bodies? Bianchi-Berthouze et al. suggested that the relationship came about as a result of increased affect.

By involving the body, the affective dimension of the experience increased, and in turn, this impacted upon engagement. Shouse (2005, p. 5) defines affect as a ‘non-conscious experience of intensity; it is a moment of unformed and unstructured potential’. Why does involving the body to a greater extent increase the affective response? And what are the implications of this relationship for the design of technologies and pedagogy?

Bianchi-Berthouze, N., Kim, W. W., & Patel, D. (2007) Does body movement engage you more in digital game play? And Why? Proceedings of the International Conference of Affective Computing and Intelligent Interaction (102-113).  

embodied metaphors and intuitive interaction

By Mona Sakr

Antle, Corness & Droumeva (2009) consider the importance of embodied metaphors in user interface design.

An embodied metaphor links elements of an embodied schema to an abstract concept. For example, we use verticality to talk about hierarchy (‘going up the career ladder’) and proximity to talk about relationships (‘they’re so close!’).

How can these types of metaphor aid the development of intuitive interfaces i.e. interfaces that offer immediate success and do what you expect them to. Antle et al. predicted that interaction in a digital environment was better supported when the body was used to control output, rather than a particular input device, and when the the link between whole-body movement and output relies on typical embodied metaphors.

To explore this, they looked at adults in an auditory environment in which whole body movements could be used to control pitch and volume. Rather than tell participants what types of movement would control sound, they observed how long participants took to determine the link between input and output. When this was done quickly, they recognised the interaction as ‘intuitive’. As predicted, common embodied metaphors (e.g. volume being related to up/down movements) helped participants to interact with the digital environment in a more intuitive way.

In conclusion, the researchers suggested that this highlighted the importance of ‘leveraging embodied metaphors in design’ (p. 252).

Antle, A. N., Corness, G., & Droumeva, M. (2009). Human-computer-intuition? Exploring the cognitive basis for intuition in embodied interaction. International Journal of Arts and Technology, 2(3), 235-254.

hand action ‘taxonomy’

By Mona Sakr

A few days ago, I talked about Streeck’s taxonomy of gesture in Gesturecraft. It’s now time to share the taxonomy of hand action we’ve developed at the lab in response to a study of the hands in scientific inquiry. This taxonomy relates particularly to scientific inquiry contexts (though it may be useful for looking at hand action in other forums of experience) and is based on ‘reading’ both the form and function of hand actions. It’s inspired by literature in the field and video analysis of students involved in inquiry learning about the behaviour of light.

1.     Ergotic movements

Ergotic movements are those that change the surrounding environment . Such movements may involve changing the position of an object, or attempting to change its physical properties. In the context of scientific inquiry, ergotic movements are necessary in order to facilitate observations of particular phenomena.

2.     Epistemic movements

Epistemic movements are those that enable an individual to know more about the physical properties of an object. While ergotic movements are designed to change the surrounding environment, epistemic movements enable better perception of the surrounding environment e.g. through feeling the texture of an object.

3.     Deictic gesture

Deictic gestures are used to point to or physically highlight objects or areas in the physical world. They may be used to draw attention to a representational field or a particular aspect within a field.

4.     Re-enactment gestures

While deictic gestures draw attention to particular parts of the environment, re-enactment gestures focus on descriptive processes and so have an added temporal dimension of expression. Through using re-enactment gestures processes that are otherwise too fast to be visible can be slowed down.

5.      Ideational gestures

While all of the actions described above relate to physical phenomena that are present, ideational gestures can be used to indicate content that is not present in any respect, like abstract ideas or previous experiences. In the context of scientific inquiry, students may wish to invoke previously learned knowledge in order to make sense of what is currently occurring. Gesture may be helpful in this because it constitutes a way of representing absent knowledge.