Augmented Harmony

“The beats emanating from Bristol’s Millennium Square announced that something was happening. The first sight was of onlookers, then a small group could be seen in front of them holding up signs which they moved to the music.

Walking in a little further it became clear what held everyone’s attention. The BBC Big Screen mounted up high was showing a live image of the group at the front, but with animated characters added amongst them, rapping out the beats.

​

This interactive performance was created by The Sancho Plan, a group of writers, musicians, animators, designers and computer programmers, in collaboration with Bristol’s Pervasive Media Studio, which hosted the project, and the BBC. As an experiment in inclusion, The Sancho Plan’s aim was to give people a chance to make music together. They needed a technology to enable those without musical training to take part, and chose augmented reality (AR): the superposition of virtual content onto live camera feeds.

​

A video camera captured the dancers (even if they started by simply holding up the signs, the participants found themselves dancing eventually). Its live image was fed directly to the screen, but also through a computer that recognised the computer-generated symbols on the signs. The computer translated each of those symbols into a small monkey-like character who sat squarely on top of it, and played a track of beats. When you move the symbol, the monkey follows. When you hide the symbol, the monkey and its beats disappear. Monkeys harmonise their beats when others are present, and an animated link appears between them.

​

There is something timelessly fascinating about seeing transformations of ourselves looking back at us – whether in a rippling pool, a hall of mirrors, or a video screen. And now, when that ‘mirror’ is interactive, and responds to us with synthesised sound and vision, the possibilities are, as we know from fairy tales, considerable. Interactive technologies for blending the virtual and physical worlds are maturing.

​

The Sancho Plan’s piece used AR to engaging and inclusive effect. There was no shortage of smiling participants, who had heard about the event or just happened to be near the square. A small sample of onlookers said it was ‘amazing’ or ‘cool’, and had appreciated the relationship between the symbols, the monkeys and the beats they rapped, even if most couldn’t say what kind of technology they were seeing in action.

This was a relatively simple use of AR, which was aided by the symbols but could in principle have used the participants’ bodies and gestures as triggers. The Sancho Plan recognises the need to explore more types of musical interaction. But the simplicity didn’t seem to matter on this occasion. The participants experimented, achieving a visual effect even when they couldn’t influence the sounds by doing so. They rotated the signs and swung them about. The groups, even those consisting of strangers, choreographed themselves spontaneously, moving around one another, stacking the signs, running, and dancing and waving to the beats. The monkeys sometimes couldn’t quite keep up with the humans. At one point a little girl without a sign watched herself dancing – with the monkeys or the people behind her, it was hard to say.

​

AR is one of many ways for artists to embed virtual content and interactivity into our physical spaces: others include ‘physical hyperlinks’ such as barcodes and electronic tags linked to content, and media overlays triggered by GPS and the other sensors found on smart phones. But artistic exploration of how to weave interactivity with physicality is in its early days. The Sancho Plan brought a musical mirror to Millennium Square. We can look forward to more ways of seeing ourselves – and the physical world around us – through the eyes of the web.”

​

– Tim Kindberg, 2010

​

About Tim Kindberg
Tim Kindberg is a computer scientist whose company, Matter 2 Media, develops technologies for urban interactivity. tim@matter2media.com

​

About Pervasive Media Studio
What is pervasive media? How can we design immersive, creative, inclusive experiences for this new form? Bringing together practitioners, researchers and users from diverse backgrounds, the Pervasive Media Studio seeks to explore these questions, establishing a globally-recognised centre of excellence in the heart of Bristol, UK.