Have your say about online reviews, ratings and rankings.
If you have five seconds, you could do us a big favour and vote for us in the EngageU awards. This is a new “European Competition for Best Innovations in University Outreach and Public Engagement”.
Just go to our submission and click on the red box in the top right corner.
And while you’re there, have a look at the other projects, too. There is some really good stuff going on.
{ 0 comments }
Lots of chatter this morning about yesterday’s Channel 4 programme “Attack of the Trip Advisors”. This is an entertaining and informative documentary about TripAdvisor reviews and ratings. As the programme snippet claims:
The British hospitality industry is under attack. Businesses are being assaulted by ever-more nit-picking and abusive reviews. It’s bad for their livelihoods and their sanity.
But it’s not the professional critics who are reviewing their hosts; a nation of virtual AA Gills and Michael Winners are using the Trip Advisor website to get their own back on hotels and restaurants.
With more than 40 million users a month, Trip Advisor is the largest and most powerful travel guide in the world.
But is it a force for good that gives the customer a voice, or an abuse of power that undermines businesses and ruins lives?
How long can Britain’s small businesses cope with relentless criticism before they pack it all in?
This Cutting Edge film reveals Britain’s most meticulous Trip Advisors and meets some of the hoteliers and restaurateurs at war with the site.
The documentary features some interesting cases and highlights many of the issues we have been dealing with in the How’s My Feedback? project. Not surprisingly, the debate has already started in the TripAdvisor section. Opinions and experiences differ wildly, so please join in and share your story. The issues clearly go beyond TripAdvisor, but it is a good case in point to discuss some timely and important problems around online reviews, ratings and rankings.
If you missed the show, you can still watch it here at Channel 4OD.
{ 0 comments }
It took a while, but the videos of the conference talks are now online. Have a look and indulge in some challenging talks around online reviews, ratings and rankings.
The slides are mostly captured in the videos. However, it might be more convenient to watch them separately. Here are the decks, which are currently available (all pdfs):
You can watch all videos also on our Vimeo account.
{ 0 comments }
Thank you very much to everyone who made yesterday’s conference a success. The day was packed with discussions, provocations, insight and — who would have thought — feedback. All sessions were recorded so we will post the videos soon. For the moment, please find below a collection of snapshots and comments.
At the heart of the conference were five talks, which offered different perspectives on the phenomenon of online reviews, ratings and rankings. After a brief introduction by Steve Woolgar and myself, Stefan Schwarzkopf drew some interesting connections between his attempts to review a hotel and ideas from political theory: “Feedback, democracy and conflicting consumption in a New York hotel: A journey from theory to micro-study, and back”. His presentation was followed by remarks from Daniel Neyland, who challenged the idea of a preexisting object of evaluation and shared a story about his own struggles with a review of a recent book of his. Next, Alex Wilkie from Goldsmiths reported on his research about “User involvement in design” and specifically the role of personas and user assemblages in the design process. In his comments, Tim Webmoor reflected on Danah Boyd’s Twitter debacle and the changing conception of the expert in evaluations.
After the lunch break, Andy Balmer reinvigorated the audience with an autoethnographic account of “Being 6.1: My life on HotorNot.com”, followed by Sally Wyatt’s remarks and provocations, which were partially delivered in the form of a t-shirt. Finally, Malcolm Ashmore reflected on the notion of reflexivity in his talk “What is it to evaluate the evaluators? A fairly formal reflexive analysis”, to which Javier Lezaun responded with some (in his own words) “unfairly informal” comments.
Christine Hine and James Munro skillfully summed up the day and offered concluding remarks, including individual ratings of speakers on a 10-point scale as well as an analysis of the speaker hotel and it’s mixed reviews.
The list of delegates was long and diverse. Besides academics, also practitioners came to Oxford, including Amazon reviewers, university administrators and social entrepreneurs from organizations like How’s My Driving Ltd, GAF Materials Corporation, MMU business school, VU University, Patient Opinion, IE Business School, BestSoftwareIndex.com, Bazaarvoice, BPP University College, ECI, London Business School, Arizona State University, University of Cambridge, University of Bedford, Research in Practice, Infosys Technologies, aporia, University of Tasmania, University of Reading, Scientific Council for Government Policy, University of Lincoln, New York University, HealthUnlocked, University of Kent, Manchester Metropolitan University, University of Leeds, UNED National Distance University of Spain, University of Leicester, The Open University, Imperial College, GHK Consulting Limited, and the London School Of Economics.
In order to get a better sense of the dynamics of evaluation, we engaged in a live experiment, using the latest development in feedback technology. Andy Balmer generously volunteered (and actually was quite keen) to participate in a real-time worm poll. Members of the audience could indicate whether they “Liked” or “Disliked” Andy’s talk at any time during the session. The individual votes were then aggregated into an evolving worm graph, displayed next to the speaker’s slides.
The worm experiment offered an excellent opportunity to experience and reflect on what it is to be evaluated as well as to evaluate. In a discussion after the session, participants shared their observations about the effort required to focus on both the talk and the worm, ethical concerns and uneasiness in anonymously judging the speaker, technical difficulties of accessing the school’s Wi-fi, the discussions among the project team about the temporal-spatial arrangement of screens, speaker slides and audience before the conference, as well as moments of gaming when participants chased each others’ movements up or down. While the worm poll may not become a standard feature in academic talks, the experiment generated some challenging questions, which would have otherwise gone unnoticed.
A special thank you is due to the outstanding developers and designers at Insight 4 Labs, who made the experiment possible. A beta version of the technology is now available for public use. So if you are interested in trying it out, have look at SocialPoll.tv. It is great fun to play with and you will certainly come up with novel applications such as voting on live TV shows.
As it is quite common these days, the conference was accompanied by a more or less lively backchannel of gossip and commentary on Twitter. Here is a selection of tweets that circulated on the day:
@scottywoodhouse @howsmyfeedback can’t wait to see the results from tomorrow’s #hmfconf
@AndyBalmer I give it a ‘4’ so far, though out of what I don’t know. “@InSIS: #hmfconf kicks off http://t.co/O2weAgf”
@InSIS Stefan Schwarzkopf – online feedback and review systems channel reviews into triviality #hmfconf
@AuntieHelen “All feedback is rubbish and leads to prostitution and bad book reviews.” Daniel Neyland. A good discussion point!
@scmward Thought provoking sessions on evaluation: rating what and for whom? The event included a real experiment using worm technology #hmfconf
@webmoor Interesting real-time experiment in academic lecture at Oxford with real-time feedback projected behind speaker using ‘worm’ rating #hmfconf
@AuntieHelen Now worming Andrew Balmer of Sheffield University… #hmfconf
@valfazel #hmfconf playing around with worms
@patientopinion “You can’t fatten a pig by weighing it” – lively debate on value of online feedback #hmfconf
@AuntieHelen Fascinating – pirate ships in 17th and 18th centuries were early democracies! #hmfconf
@valfazel #hmfconf Javier Lezaun “In ethnomethodological research there are no rhetorical questions”
Last not least, an important actor at the conference was of course the How’s My Feedback? prototype. If you haven’t done so, please drop by and give us (or others) feedback:
https://www.howsmyfeedback.org
Many thanks to the ESRC, the Institute for Science, Innovation and Society and our project partner Patient Opinion for their generous and ongoing support.
{ 0 comments }
It’s only a week until the How’s My Feedback? conference in Oxford. We now have a preliminary programme for the day with a great line-up of speakers, talks and topics. Specifically, watch out for the world premiere of a brand-new feedback technology — and of course the prototype, which we will share with you shortly.
If you haven’t signed up yet, please do so as soon as possible. There are only a few spaces left.
{ 1 comment }
It’s been a few weeks since our second expert workshop, so here comes a brief update on what has happened since:
On a very different occasion, I presented the project at the Harvard-MIT-Yale Cyberscholar Working Group at the MIT Media Lab. This was a great opportunity to get feedback from a very diverse crowd of people, including media designers, HCI researchers, lawyers and social scientists. There was also a second presentation by Nick Bramble, which very nicely highlighted the important legal issue of third-party liability for content posted on review and rating websites.More updates soon. Again, don’t forget to register for the conference.
{ 0 comments }
Registration is now open for How’s My Feedback? – The Technology and Politics of Evaluation, a one-day international conference at the Institute for Science, Innovation and Society, Oxford University.
Tuesday, 28 June 2011
9.00 – 17.30
Saïd Business School, University of Oxford
About the conference:
There is hardly anything these days that is not being evaluated on the web. Books, dishwashers, lawyers, teachers, health services, ex-boyfriends, haircuts, prostitutes and websites are just some examples targeted by novel review, rating and ranking schemes. Used in an increasing number of areas, these schemes facilitate public assessment by soliciting and aggregating feedback and distributing it as comments, ranks, scales and stories. While some have greeted this development as an innovative way of fostering transparency, accountability and public engagement, others have criticized the forced exposure and alleged lack of accuracy and legitimacy, pointing to the potentially devastating consequences of negative evaluations.
Now research is under way to tackle these issues head-on and evaluate the various types of review, rating and ranking schemes in a collaborative design experiment. Under the title ‘How’s my feedback?’, a group of experts, including designers, managers, reviewers, policy-makers, consumer spokespeople, academics and users are currently exploring the idea of a website that allows users to publicly assess their experience with review and rating schemes – a feedback website for feedback websites.
The goal of the conference is to reflect on this process and the emerging prototype. How are we to judge the effectiveness of these schemes? What modes of governance are implicated in their operation? What strategies and methodologies are employed in their development, maintenance and use? How successful is this project as a design intervention? What is it to evaluate the evaluators – and will this business ever end?
Speakers include:
Organisers:
Malte Ziewitz and Steve Woolgar, University of Oxford, in cooperation with James Munro, Patient Opinion
Registration:
The event is free of charge, but registration is required: REGISTER HERE
Project website: https://www.howsmyfeedback.org/
Twitter: http://twitter.com/howsmyfeedback/
Download poster: https://www.howsmyfeedback.org/poster.pdf
How to find us: http://goo.gl/maps/hLW8
For more information, contact insisevents@sbs.ox.ac.uk.
The conference is generously supported by an ESRC Knowledge Exchange Small Grant and the Institute for Science, Innovation and Society.
UPDATE 27/5/2011: We just moved registration to a new site. If you already registered, no worries. You are still signed up.
{ 2 comments }
Here are some quick impressions from yesterday’s second expert workshop. Again, we met in London at the offices of Which? — this time to dicuss and imagine the prototype we are supposed to built over the next couple of weeks.
Building on insights from the first expert workshop, we focused on a number of rapid design exercises. We split up into two groups and equipped ourselves with pens, paper, flipcharts and sticky notes. Under James’ guidance, we started by devising personas, i.e. concrete individuals who might use How’s my feedback?, and then sketched their journeys to and through the website. This also led us to consider some of the functionalities of the prototype and rethink its scope and purpose.
 |
 |
 |
 |
As expected, the design process was much messier, but also more interesting than any textbook could have taught. A few examples:
Despite (or perhaps because of) these difficulties, we managed to explore a range of cases, uses and functionalities. Among other things, we discovered new aspects like “the potential to cause trouble” or the possibility of becoming a clearinghouse that might help inquiry and research more than decision-making. It is still a long way to go, but discussing the issues with a terrific group of experts is a more than worthwhile (and fun!) part of the journey.
Many thanks to everyone who contributed, especially Lorraine Aziz, Melanie Dowding, Peter Harris, Chris Emmins, Kirsten Guthrie, Helen Hancox, Dixon Jones, Noortje Marres, Stefan Schwarzkopf, Paul Smith, Marcus Taylor, Esther Vicente, Jonathan Wolf.
{ 0 comments }
How to make sense of the recent explosion of online review and rating schemes? What are the issues and how can we think about them? At the first expert workshop, I gave a brief introduction to the ‘How’s my feedback?’ project and the puzzles that have got us thinking. This is a brief overview of the argument, which was meant to open up the discussion. Feel free to download the slides, though the text below might give you a better idea of what this was about.
The presentation starts with an obviously incomplete map of existing web-based review and rating schemes. The purpose of this was not to provide the ultimate taxonomy, but rather emphasise the wide range of areas in which online feedback has been implemented. There are endless ways to order these schemes: who runs them (government, business, third sector), which inputs are required (direct through reviews and ratings, indirect through tracing and tracking), what methodology is used (aggregation, statistical modelling, algorithms), how are results presented (fulltext, snippets; ratios, scores, stars) and organised (chronological order, ranked, filtered), etc. However, what all of them have in common is that they use electronic technologies to generate and mobilise public evaluations.
Now, it is interesting to think about the ways of social ordering these schemes bring about. There is a long history in this line of thinking about evaluation technologies, especially on accounting, audit or polling devices. However, what is interesting about web-based review and rating schemes is that they tend to oscillate between three common claims: (1) transparency: feedback makes hidden qualities transparent and allows better informed decisions; (2) accountability: the publicity of evaluations holds their targets to account, thus facilitating social control, and (3) participation: the capacity to involve a large number of people and foster engagement, empowerment and scale.
Against this backdrop, different schemes emphasise different aspects. While a search engine like Google may run on a transparency ticket (“organize the world’s information and make it universally accessible and useful”), it also invokes ideas of participation (counting links as votes) and accountability (if you want to rank high, produce ‘relevant’ content). Conversely, a website like MyPolice may aim to improve police services (“a neutral space where you can tell your story secure in the knowledge that the people who count will read it”; accountability) by making people’s experiences visible (“understand what the public needs and wants”) and giving everyone a voice (“You pay for your police service. … You’re the one who matters.”).
However, while this simple model may be helpful for understanding the range of feedback schemes and their corresponding claims, the everyday practices of reviewing and rating look quite different:
1. Users sometimes behave strangely: Technology tends to be designed with a certain user in mind. In a sense, it ‘configures’ the user as Steve argued previously. So what happens if these users talk back and use a platform in ways we did not anticipate (or appreciate)? What assumptions about users come with certain schemes – and how do they work out in practice?
2. Not everybody wants to be empowered: A key issue for a lot of schemes is actually attracting a sufficient volume of feedback. For example, a common sight these days is the invitation to “Be the first to write a review!”. What does this tell us about calls for ‘democracy’ and ‘participation’?
3. There is more than one truth: Take a look at Phil’s exchange with a reviewer on TripAdvisor. Who is the expert in this case: the person claiming to be the guest or the owner responding to the feedback? What counts as ‘valid’ knowledge – and to whom?
4. Not all publicity is good publicity: A common set of concerns these days revolve around the risk of libel and exposure. For example, what is it to be named and called a “habitual convict”, “liar” and a “drug addict” as it happens on DontDateHimGirl? A certain degree of exposure may be a nifty way to attract more people to your website. But where does ‘fair’ marketing end and where does ‘unfair’ libel start?
5. Metrics become targets: And not just metrics, also stories, commments and other forms of feedback. Is it a problem when you can buy a review for $5? What if you organise your friends to express their protest against landmines by participating in a poll on the New Tork Times website? What counts as ‘legitimate’ or ‘ethical’ engagement?
6. Resistance is tricky: Finally, there are increasing concerns about due process. Who holds the evaluators to account — and how? A number of strategies can be observed, including legal action (i.e. invoking a ‘higher’ evaluation scheme), counter schemes (see, e.g. Guestscan as a response to hotel reviews) and watchblogs. So how to evaluate the evaluators — and does this business ever stop?
Overall, it seems that many schemes defy the simple logic of the underlying models. Web-based evaluation is not just about ‘data’ and ‘information’, but deeply embedded in and constitutive of social relations. Qualities are not simply measured or described, but negotiated and performed in an ongoing and messy social process. As a result, public evaluation is never innocent, but highly political with sometimes devastating consequences for those reviewed, ranked and rated. And this is exactly where How’s my feedback? comes in — and our task to design and tinker with a prototype that allows people publicly to evaluate their experiences with review and rating schemes.
{ 0 comments }
The first expert workshop is over, and the ideas are piling up on our desks. The range and richness of views and insights was absolutely amazing. This promises to be a great project, and we are already looking forward to the second workshop on 11 April.
The openness and energy of the expert group was quite impressive. Although the workshops bring together a rather diverse group of people, this did not prevent anyone from jumping right into the discussion, sharing and challenging expertise in marketing, monetizing, facilitating, soliciting, moderating, preventing, evaluating, giving and receiving feedback online. We are currently working on a detailed summary to prepare the ground for the design work in the second workshop. So for the moment, this is just a brief overview.
 |
 |
 |
 |
Introduction and background: In an attempt to introduce the project, I mapped the current landscape of web-based review and rating schemes and sketched six puzzles that had got us thinking in the project team. We will write more about this soon.
Stories from the field: Among the highlights of the afternoon were certainly the talks of three expert group members, who had volunteered to kick us off.
Group work: In the last hour of the workshop, we split up into two groups and discussed four questions based on the presentations: What is online feedback for? What are the benefits — and to whom? What are the harms — and to whom? And what counts as a “good” and a “bad” feedback scheme? Again, we are currently working on a summary of the discussion. Quite a daunting task, but an essential step on our way to the prototype.
Many thanks to David Albury, Sarah Drinkwater, Peter Durward Harris, William Heath, Helen Hancox, Harry Metcalfe, John Robinson, Stefan Schwarzkopf, Jason Smith, Marcus Taylor and Elizabeth Forrester at Which? for making the first Expert Workshop a success.
{ 2 comments }
