Thank you very much to everyone who made yesterday’s conference a success. The day was packed with discussions, provocations, insight and — who would have thought — feedback. All sessions were recorded so we will post the videos soon. For the moment, please find below a collection of snapshots and comments.
1. The speakers
At the heart of the conference were five talks, which offered different perspectives on the phenomenon of online reviews, ratings and rankings. After a brief introduction by Steve Woolgar and myself, Stefan Schwarzkopf drew some interesting connections between his attempts to review a hotel and ideas from political theory: “Feedback, democracy and conflicting consumption in a New York hotel: A journey from theory to micro-study, and back”. His presentation was followed by remarks from Daniel Neyland, who challenged the idea of a preexisting object of evaluation and shared a story about his own struggles with a review of a recent book of his. Next, Alex Wilkie from Goldsmiths reported on his research about “User involvement in design” and specifically the role of personas and user assemblages in the design process. In his comments, Tim Webmoor reflected on Danah Boyd’s Twitter debacle and the changing conception of the expert in evaluations.
After the lunch break, Andy Balmer reinvigorated the audience with an autoethnographic account of “Being 6.1: My life on HotorNot.com”, followed by Sally Wyatt’s remarks and provocations, which were partially delivered in the form of a t-shirt. Finally, Malcolm Ashmore reflected on the notion of reflexivity in his talk “What is it to evaluate the evaluators? A fairly formal reflexive analysis”, to which Javier Lezaun responded with some (in his own words) “unfairly informal” comments.
Christine Hine and James Munro skillfully summed up the day and offered concluding remarks, including individual ratings of speakers on a 10-point scale as well as an analysis of the speaker hotel and it’s mixed reviews.
2. The audience
The list of delegates was long and diverse. Besides academics, also practitioners came to Oxford, including Amazon reviewers, university administrators and social entrepreneurs from organizations like How’s My Driving Ltd, GAF Materials Corporation, MMU business school, VU University, Patient Opinion, IE Business School, BestSoftwareIndex.com, Bazaarvoice, BPP University College, ECI, London Business School, Arizona State University, University of Cambridge, University of Bedford, Research in Practice, Infosys Technologies, aporia, University of Tasmania, University of Reading, Scientific Council for Government Policy, University of Lincoln, New York University, HealthUnlocked, University of Kent, Manchester Metropolitan University, University of Leeds, UNED National Distance University of Spain, University of Leicester, The Open University, Imperial College, GHK Consulting Limited, and the London School Of Economics.
3. The worm experiment
In order to get a better sense of the dynamics of evaluation, we engaged in a live experiment, using the latest development in feedback technology. Andy Balmer generously volunteered (and actually was quite keen) to participate in a real-time worm poll. Members of the audience could indicate whether they “Liked” or “Disliked” Andy’s talk at any time during the session. The individual votes were then aggregated into an evolving worm graph, displayed next to the speaker’s slides.
The worm experiment offered an excellent opportunity to experience and reflect on what it is to be evaluated as well as to evaluate. In a discussion after the session, participants shared their observations about the effort required to focus on both the talk and the worm, ethical concerns and uneasiness in anonymously judging the speaker, technical difficulties of accessing the school’s Wi-fi, the discussions among the project team about the temporal-spatial arrangement of screens, speaker slides and audience before the conference, as well as moments of gaming when participants chased each others’ movements up or down. While the worm poll may not become a standard feature in academic talks, the experiment generated some challenging questions, which would have otherwise gone unnoticed.
A special thank you is due to the outstanding developers and designers at Insight 4 Labs, who made the experiment possible. A beta version of the technology is now available for public use. So if you are interested in trying it out, have look at SocialPoll.tv. It is great fun to play with and you will certainly come up with novel applications such as voting on live TV shows.
4. The reactions
As it is quite common these days, the conference was accompanied by a more or less lively backchannel of gossip and commentary on Twitter. Here is a selection of tweets that circulated on the day:
@scottywoodhouse @howsmyfeedback can’t wait to see the results from tomorrow’s #hmfconf
@AndyBalmer I give it a ‘4’ so far, though out of what I don’t know. “@InSIS: #hmfconf kicks off http://t.co/O2weAgf”
@InSIS Stefan Schwarzkopf – online feedback and review systems channel reviews into triviality #hmfconf
@AuntieHelen “All feedback is rubbish and leads to prostitution and bad book reviews.” Daniel Neyland. A good discussion point!
@scmward Thought provoking sessions on evaluation: rating what and for whom? The event included a real experiment using worm technology #hmfconf
@webmoor Interesting real-time experiment in academic lecture at Oxford with real-time feedback projected behind speaker using ‘worm’ rating #hmfconf
@AuntieHelen Now worming Andrew Balmer of Sheffield University… #hmfconf
@valfazel #hmfconf playing around with worms
@patientopinion “You can’t fatten a pig by weighing it” – lively debate on value of online feedback #hmfconf
@AuntieHelen Fascinating – pirate ships in 17th and 18th centuries were early democracies! #hmfconf
@valfazel #hmfconf Javier Lezaun “In ethnomethodological research there are no rhetorical questions”
5. The prototype
Last not least, an important actor at the conference was of course the How’s My Feedback? prototype. If you haven’t done so, please drop by and give us (or others) feedback:
Many thanks to the ESRC, the Institute for Science, Innovation and Society and our project partner Patient Opinion for their generous and ongoing support.