How to make sense of the recent explosion of online review and rating schemes? What are the issues and how can we think about them? At the first expert workshop, I gave a brief introduction to the ‘How’s my feedback?’ project and the puzzles that have got us thinking. This is a brief overview of the argument, which was meant to open up the discussion. Feel free to download the slides, though the text below might give you a better idea of what this was about.
The presentation starts with an obviously incomplete map of existing web-based review and rating schemes. The purpose of this was not to provide the ultimate taxonomy, but rather emphasise the wide range of areas in which online feedback has been implemented. There are endless ways to order these schemes: who runs them (government, business, third sector), which inputs are required (direct through reviews and ratings, indirect through tracing and tracking), what methodology is used (aggregation, statistical modelling, algorithms), how are results presented (fulltext, snippets; ratios, scores, stars) and organised (chronological order, ranked, filtered), etc. However, what all of them have in common is that they use electronic technologies to generate and mobilise public evaluations.
Now, it is interesting to think about the ways of social ordering these schemes bring about. There is a long history in this line of thinking about evaluation technologies, especially on accounting, audit or polling devices. However, what is interesting about web-based review and rating schemes is that they tend to oscillate between three common claims: (1) transparency: feedback makes hidden qualities transparent and allows better informed decisions; (2) accountability: the publicity of evaluations holds their targets to account, thus facilitating social control, and (3) participation: the capacity to involve a large number of people and foster engagement, empowerment and scale.
Against this backdrop, different schemes emphasise different aspects. While a search engine like Google may run on a transparency ticket (“organize the world’s information and make it universally accessible and useful”), it also invokes ideas of participation (counting links as votes) and accountability (if you want to rank high, produce ‘relevant’ content). Conversely, a website like MyPolice may aim to improve police services (“a neutral space where you can tell your story secure in the knowledge that the people who count will read it”; accountability) by making people’s experiences visible (“understand what the public needs and wants”) and giving everyone a voice (“You pay for your police service. … You’re the one who matters.”).
However, while this simple model may be helpful for understanding the range of feedback schemes and their corresponding claims, the everyday practices of reviewing and rating look quite different:
1. Users sometimes behave strangely: Technology tends to be designed with a certain user in mind. In a sense, it ‘configures’ the user as Steve argued previously. So what happens if these users talk back and use a platform in ways we did not anticipate (or appreciate)? What assumptions about users come with certain schemes – and how do they work out in practice?
2. Not everybody wants to be empowered: A key issue for a lot of schemes is actually attracting a sufficient volume of feedback. For example, a common sight these days is the invitation to “Be the first to write a review!”. What does this tell us about calls for ‘democracy’ and ‘participation’?
3. There is more than one truth: Take a look at Phil’s exchange with a reviewer on TripAdvisor. Who is the expert in this case: the person claiming to be the guest or the owner responding to the feedback? What counts as ‘valid’ knowledge – and to whom?
4. Not all publicity is good publicity: A common set of concerns these days revolve around the risk of libel and exposure. For example, what is it to be named and called a “habitual convict”, “liar” and a “drug addict” as it happens on DontDateHimGirl? A certain degree of exposure may be a nifty way to attract more people to your website. But where does ‘fair’ marketing end and where does ‘unfair’ libel start?
5. Metrics become targets: And not just metrics, also stories, commments and other forms of feedback. Is it a problem when you can buy a review for $5? What if you organise your friends to express their protest against landmines by participating in a poll on the New Tork Times website? What counts as ‘legitimate’ or ‘ethical’ engagement?
6. Resistance is tricky: Finally, there are increasing concerns about due process. Who holds the evaluators to account — and how? A number of strategies can be observed, including legal action (i.e. invoking a ‘higher’ evaluation scheme), counter schemes (see, e.g. Guestscan as a response to hotel reviews) and watchblogs. So how to evaluate the evaluators — and does this business ever stop?
Overall, it seems that many schemes defy the simple logic of the underlying models. Web-based evaluation is not just about ‘data’ and ‘information’, but deeply embedded in and constitutive of social relations. Qualities are not simply measured or described, but negotiated and performed in an ongoing and messy social process. As a result, public evaluation is never innocent, but highly political with sometimes devastating consequences for those reviewed, ranked and rated. And this is exactly where How’s my feedback? comes in — and our task to design and tinker with a prototype that allows people publicly to evaluate their experiences with review and rating schemes.
{ 0 comments }
Group work: In the last hour of the workshop, we split up into two groups and discussed four questions based on the presentations: What is online feedback for? What are the benefits — and to whom? What are the harms — and to whom? And what counts as a “good” and a “bad” feedback scheme? Again, we are currently working on a summary of the discussion. Quite a daunting task, but an essential step on our way to the prototype.
