subconscious consensus: Playing Games with Collaborative control
The Pong Experiment
In 1991, Loren Carpenter (co-founder of Pixar), ran an experiment at SIGGRAPH. People entered a theatre to find paddles had been left on their seats, with one green side and one red side. As they held up the paddles, red and green squares appeared on a screen at the front of the theatre. Audience members were able to identify their own paddle in the crowd on the screen.
Then a game of Pong appeared and the audience came to realise that they had been split into two halves, with each team controlling one of the players in the game collaboratively, using their paddles – green for up, red for down.
Audience plays collaborative Pong at SIGGRAPH ‘91, from this video
Each group operated as a cohesive entity to control their player in the Pong game. In the BBC documentary ‘All Watched Over by Machines of Loving Grace’, Loren describes this effect:
“They’re all acting as individuals, because each one of them can decide what they’re going to do. There’s an order that emerges that gives them kind of like an amoeba like effect where they surge and they play. I wanted to see if no hierarchy existed at all, what would happen? They formed a kind of a subconscious consensus.”
I found this fascinating and wanted to recreate Loren Carpenter’s Pong experiment to learn from going through the process and see for myself how the audience reacts.
Technical Research
From a technical perspective, it’s pretty mind blowing that Carpenter achieved this in 1991. That’s long before games consoles with tracking controllers like Wii or Playstation Move existed and there were no libraries readily available for computer vision or blob tracking.
Carpenter’s patent for the technology reveals some of the secrets. Physically, the system relies on a light mounted next to the camera which reflects off high vis material on the audience paddles, making them easier to identify. In terms of the software, the entire thing was coded from scratch in C.
Diagram from Loren Carpenter’s patent, ‘Method and apparatus for audience participation by electronic imaging’
Prototyping
Luckily for me, these days, Processing, OpenCV and Youtube tutorials are here to help. I experimented with blob detection using this tutorial by Daniel Shiffman. With a little adjustment of the thresholds, and some coloured cards, I was able get reliable results.
There are some false positives, and I expect that in a large audience this would happen often, so I ordered some reflective tape to experiment with Carpenter’s method.
I thought I would need a particular type of light and a dark room to get this to work at all. In fact, using my phone’s torch light next to my Canon 7D in the normally lit auditorium worked really well.
The Friend test
Next I coded up a Pong game, and took advantage of my friend’s Pancake Day gathering to playtest. The camera feed is split into two halves, creating two teams, each controlling one of the paddles. On the left in the video below is my “debug” view and the right is the game itself.
Early Observations
People enjoyed playing and got very competitive. One friend asked whether it was fair play to lean over and put his card into the other team’s space to mess them up. My goal is to explore collective feelings more than competition so I quickly coded up a rally counter and displayed a current rally count and top score for the longest rally so far. This made it more of a collaborative effort across the whole group rather than a competitive one between two teams.
Unsurprisingly, people didn’t enjoy the light shining in their face and we reverted to plain colours rather than reflection. I was able to control the space to avoid false positives (removing a red cushion and a red pair of scissors) so this worked fine.
In general people said that the game felt very responsive, and they did feel like they were having an influence on the movement of the pong paddle, which was encouraging to hear. The biggest issue was that it is hard to make the paddle stay still. If there are more red cards than yellow, the paddle moves down; more yellow than red, it moves up, it rarely stays still and in an odd numbered team, it never does. I adjusted the code so that a larger majority was needed to make the paddle move. Everyone said it felt much less responsive. In a 3-person team, everyone had to have their card in agreement to make the paddle move and we soon reverted back. Perhaps in a larger group this would work better.
larger tests
Up until now I’d been using hastily constructed cardboard paddles but for my next tests I made some laser cut ones to improve the overall audience experience.
I also experimented with different lighting setups and found that a desk lamp provided a more dispersed light, enabling paddles to be picked up further away, while also being less blinding for audience members than my phone light had been.
Having tested as much as I could with paddles taped to the auditorium steps, I recruited some fellow students to play a game of Pong.
Then, the following week, I ran a larger test with a more varied audience including people I didn’t know and who had no background knowledge of the project. In this test they played Pong, as well as a game based on Chrome’s T-Rex Dinosaur game but with an elephant jumping over mice, and one based on Flappy Bird but with a rocket. In Pong the audience were in two teams, and for the other games the whole audience played as one team.
Observations
In a larger group, control could be more finely tuned. In Pong, 60% of cards had to be one colour for the paddle to move, and if there were over 90% then it moved faster. The speed of the paddle was initially mapped too slowly but I quickly increased it in the code and reloaded the game. This kind of fine control is hard to test for without an actual crowd and it did cause some initial frustration for players - showing the importance of these kinds of tests.
Some participants try to control the group by shouting out instructions, while others stay quiet. In the jumping game, the elephant jumps when there are over 60% yellow cards, meaning participants need to flip their paddles just at the right moment. Different participants can be heard shouting “now! now!” at different moments, trying to influence the rest of the group.
In the Rocket game (inspired by Flappy Bird), the y-position of the rocket directly correlates to the percentage of red versus yellow paddles. There was much less shouting. There is no point shouting out “red!”, because if everyone goes to red, then the rocket will crash into the wall. This wasn’t explained to the audience but the lack of shouting implies to me that they intuited something about the way it was controlled. A focused hush came over the crowd and this game was the most interesting for me to watch, because it’s more purely collaborative.
When I spoke to participants afterwards, most of them were very focused on how I could alter the system to enable them to achieve higher scores. They offered suggestions like having a second screen set up so they could see what colour everyone else’s paddles were. It will be worth thinking about creating interactions that aren’t so goal-focused, to draw people away from a success/failure mindset and hopefully more into the feeling of acting as a group.
I was worried that in a larger group, people would find the dissolution of their own influence frustrating but this wasn’t reported to me. If anything, I think people enjoyed the absurdity of it. There was a lot of laughter and “enjoyable frustration” when they failed.
Gameplay improved over time. With the Rocket game, the audience played around 8 games with no points at all and I was about to move on, accepting it was too difficult. Then suddenly something clicked, they gained control and managed to travel past 3 planets, and then in the next game they were similarly successful, almost like they’d tuned into each other.
The attention span of the audience was shorter than I’d hoped, as they’d just sat through an unrelated 90 minute lecture when we started this test. There were also some technical issues, one participant’s red jumper was getting picked up sometimes, and it would be good to test this system in a more controlled environment where I could change the lighting, or just to move on from the 90’s technology and implement something with modern controllers.
Despite various difficulties and technical hitches, people enjoyed playing these games, and it was super interesting to see how they responded. Audience members started chatting and engaging (even those who didn’t know each other), which was great to see. There is huge potential for multi-user interactions to facilitate connection between audiences.
