ratings

Conference programme out now!

by Admin on June 21, 2011

It’s only a week until the How’s My Feedback? conference in Oxford. We now have a preliminary programme for the day with a great line-up of speakers, talks and topics. Specifically, watch out for the world premiere of a brand-new feedback technology — and of course the prototype, which we will share with you shortly.

How's My Feedback? Conference Programme

If you haven’t signed up yet, please do so as soon as possible. There are only a few spaces left.

{ 1 comment }

Project update: What’s new in May?

by Admin on May 22, 2011

It’s been a few weeks since our second expert workshop, so here comes a brief update on what has happened since:

  • As you might have noticed, the announcement for the one-day conference on 28 June is out. We have a stellar line-up of speakers, including Malcom Ashmore, Andrew Balmer, Alex Wilkie, Ian Stronach and Stefan Schwarzkopf. While they all come from academic backgrounds, they promise to give some interesting (and, I have been told, entertaining) feedback on the project and issues of online evaluation more generally. If you would like to participate, please register soon. The event is free, but places are limited.
  • We also have been around a bit to talk about the project. Two occasions have been particularly interesting. On 19 April 2011, I participated in a panel discussion at the Internet Freedom Conference in Strasbourg organised by the Council of Europe. ‘Multistakeholderism’ is a popular idea in this context, and the group was particularly interested in the potential of web-based reviews and ratings for fostering participation and engagement in policy-making. The video of Panel 6 is still online.
  • Harvard-MIT-Yale Cyberscholar Working GroupOn a very different occasion, I presented the project at the Harvard-MIT-Yale Cyberscholar Working Group at the MIT Media Lab. This was a great opportunity to get feedback from a very diverse crowd of people, including media designers, HCI researchers, lawyers and social scientists. There was also a second presentation by Nick Bramble, which very nicely highlighted the important legal issue of third-party liability for content posted on review and rating websites.
  • Of course, we have also been working on the prototype. It has been far from easy, given the shoestring budget and tight timeframe we are on. However, while things are moving slowly, they are moving and we hope to have something to tinker with soon. If you think you can contribute anything to the process from design skills to a developer brain, it’s not too late.
  • Finally, a lot of people got in touch and offered their support or simply showed interest in the project. In this context, have a look at other initiatives, such as Eric Goldman’s and Jason Schultz’s new project Doctored Reviews that aims to help people deal with restrictions on online patient reviews.

More updates soon. Again, don’t forget to register for the conference.

{ 0 comments }

Registration is now open for How’s My Feedback? – The Technology and Politics of Evaluation, a one-day international conference at the Institute for Science, Innovation and Society, Oxford University.

Tuesday, 28 June 2011
9.00 – 17.30
Saïd Business School, University of Oxford

About the conference:

There is hardly anything these days that is not being evaluated on the web. Books, dishwashers, lawyers, teachers, health services, ex-boyfriends, haircuts, prostitutes and websites are just some examples targeted by novel review, rating and ranking schemes. Used in an increasing number of areas, these schemes facilitate public assessment by soliciting and aggregating feedback and distributing it as comments, ranks, scales and stories. While some have greeted this development as an innovative way of fostering transparency, accountability and public engagement, others have criticized the forced exposure and alleged lack of accuracy and legitimacy, pointing to the potentially devastating consequences of negative evaluations.

Now research is under way to tackle these issues head-on and evaluate the various types of review, rating and ranking schemes in a collaborative design experiment. Under the title ‘How’s my feedback?’, a group of experts, including designers, managers, reviewers, policy-makers, consumer spokespeople, academics and users are currently exploring the idea of a website that allows users to publicly assess their experience with review and rating schemes – a feedback website for feedback websites.

The goal of the conference is to reflect on this process and the emerging prototype. How are we to judge the effectiveness of these schemes? What modes of governance are implicated in their operation? What strategies and methodologies are employed in their development, maintenance and use? How successful is this project as a design intervention? What is it to evaluate the evaluators – and will this business ever end?

Speakers include:

Organisers:

Malte Ziewitz and Steve Woolgar, University of Oxford, in cooperation with James Munro, Patient Opinion

Registration:

The event is free of charge, but registration is required: REGISTER HERE

Project website: https://www.howsmyfeedback.org/
Twitter: http://twitter.com/howsmyfeedback/
Download poster: https://www.howsmyfeedback.org/poster.pdf
How to find us: http://goo.gl/maps/hLW8

For more information, contact insisevents@sbs.ox.ac.uk.

The conference is generously supported by an ESRC Knowledge Exchange Small Grant and the Institute for Science, Innovation and Society.

UPDATE 27/5/2011: We just moved registration to a new site. If you already registered, no worries. You are still signed up.

{ 2 comments }

Here are some quick impressions from yesterday’s second expert workshop. Again, we met in London at the offices of Which? — this time to dicuss and imagine the prototype we are supposed to built over the next couple of weeks.

Building on insights from the first expert workshop, we focused on a number of rapid design exercises. We split up into two groups and equipped ourselves with pens, paper, flipcharts and sticky notes. Under James’ guidance, we started by devising personas, i.e. concrete individuals who might use How’s my feedback?, and then sketched their journeys to and through the website. This also led us to consider some of the functionalities of the prototype and rethink its scope and purpose.

A cup of pens. Jonathan and Stefan tracing user journeys. Dixon making a point.

As expected, the design process was much messier, but also more interesting than any textbook could have taught. A few examples:

  • Working with constraints: Although it was tempting to assume a world of unlimited resources, we constantly had to remind ourselves of the shoestring budget we are on. For example, why not negotiate cooperations with major operators of feedback platforms? Why not devise an algorithm that would crawl, collect and crunch transactional data to arrive at useful feedback scores for specific platforms? Or why not have an independent team of researchers explore and evaluate the schemes according to a set of universal principles? While all these ideas seemed great, we would hardly be able to pursue any of them with the time and resources at our disposal. So we were forced to think creatively about less expensive alternatives.
  • Increasing ambiguity: Another interesting phenomenon we encountered can be described in terms of Mackenzie’s certainty trough, a schematic representation of how certainty about an established technology might be distributed. Put simply, what looks like a clear case from a reasonable distance is getting more and more uncertain and ambiguous the deeper you become involved. This happened — among other things — when we dived into the website and its possible uses and features. When thinking about ‘How’s my feedback?’ in terms of quite specific situations, we sometimes feared to loose focus and needed to remind ourselves of the reasons we set out to do this.
  • Configuring users: Designing a website requires a lot of imagination. Specifically, imagining individual users turned out to be more difficult than expected. We often slipped back into talking about what ‘users’ or ‘consumers’ generally want, how smart or dumb they are and what problems they have. Built into these claims were a number of rather strong assumptions that did well in strengthening our respective arguments, but less so in helping us think from the perspective of ‘actual’ users. So thank you very much to the imaginary “Caroline (57)”, “John (35)” and “Fantom (43)” for keeping us on track.
  • Steering clear of ‘ideal types’: Another challenge was to steer clear of ideal conceptions of existing schemes and use them as the only reference point for ‘How’s my feedback?’. For example, we often found ourselves inadvertently referring to a ‘Tripadvisor-like’ system, even though the challenge is certainly much broader. This also brought up the critical question of the target of the scheme: what kind of object is ‘feedback’ and how can it usefully be assessed?

Despite (or perhaps because of) these difficulties, we managed to explore a range of cases, uses and functionalities. Among other things, we discovered new aspects like “the potential to cause trouble” or the possibility of becoming a clearinghouse that might help inquiry and research more than decision-making. It is still a long way to go, but discussing the issues with a terrific group of experts is a more than worthwhile (and fun!) part of the journey.

Many thanks to everyone who contributed, especially Lorraine Aziz, Melanie Dowding, Peter Harris, Chris Emmins, Kirsten Guthrie, Helen Hancox, Dixon Jones, Noortje Marres, Stefan Schwarzkopf, Paul Smith, Marcus Taylor, Esther Vicente, Jonathan Wolf.

{ 0 comments }

The first expert workshop is over, and the ideas are piling up on our desks. The range and richness of views and insights was absolutely amazing. This promises to be a great project, and we are already looking forward to the second workshop on 11 April.

The openness and energy of the expert group was quite impressive. Although the workshops bring together a rather diverse group of people, this did not prevent anyone from jumping right into the discussion, sharing and challenging expertise in marketing, monetizing, facilitating, soliciting, moderating, preventing, evaluating, giving and receiving feedback online. We are currently working on a detailed summary to prepare the ground for the design work in the second workshop. So for the moment, this is just a brief overview.

John guides the group through NHS Choices. Coffee keeps feedback experts going.

Introduction and background: In an attempt to introduce the project, I mapped the current landscape of web-based review and rating schemes and sketched six puzzles that had got us thinking in the project team. We will write more about this soon.

Stories from the field: Among the highlights of the afternoon were certainly the talks of three expert group members, who had volunteered to kick us off.

  • First up was Jason Smith, Client Partner at Bazaarvoice, who talked about his experience with designing and managing feedback systems for big companies like Argos and Expedia. Bazaarvoice has only been around for a bit more than five years, but already generated 196,224,118,952 conversations across its platforms (and counting). Jason showed how this data can be crunched and analysed with custom-made tools. This includes an early warning system for identifying product failures or capturing the sentiment of customers to improve marketing strategies.
  • Next, Peter Harris provided us with a fascinating inside view of what it takes to be a top reviewer on Amazon. A quick look at the top reviewer table confirms that Peter knows what he is talking about. Interestingly, it turned out that it is not always a blessing to lead the lot. Being a no. 1 reviewer comes with its own pitfalls and politics, such as receiving more critical comments or being offered free products that do not interest him. Other aspects Peter covered in his talk concerned the differences between country versions of the Amazon website, the implications of the change from the old to the new ranking system and the many ways in which reviewers interact among each other through forums, e-mails and comments.
  • Finally, John Robinson, User-generated Content Lead at NHS Choices, gave a guided tour through the comment functionalities of the government-run NHS Choices website. John talked about the challenges of designing a scheme that meets the expectations of both policy-makers and users. One example is the difficult question of moderation: what is OK to mention on a public health website and what might interfere with complaints procedures or even legal proceedings? How do negative comments affect small GP practices as opposed to big hospitals? And how to make sure that changes resulting from online reviews are sufficiently visible to the patient and the feedback loop is closed?

William and Harry sort things out at the flipchart.Group work: In the last hour of the workshop, we split up into two groups and discussed four questions based on the presentations: What is online feedback for? What are the benefits — and to whom? What are the harms — and to whom? And what counts as a “good” and a “bad” feedback scheme? Again, we are currently working on a summary of the discussion. Quite a daunting task, but an essential step on our way to the prototype.

Many thanks to David Albury, Sarah Drinkwater, Peter Durward Harris, William Heath, Helen Hancox, Harry Metcalfe, John Robinson, Stefan Schwarzkopf, Jason Smith, Marcus Taylor and Elizabeth Forrester at Which? for making the first Expert Workshop a success.

{ 2 comments }

Sneak preview of the first Expert Workshop

by Admin on March 28, 2011

Time has flown by since the project started last fall, and the first expert workshop is only two days away. Everyone who has ever organised such an event (or, in fact, any event) knows the strange tension between happy anticipation and panic attacks caused by unwell speakers or the discovery of an overlooked e-mail at the bottom of the inbox.

Which? Head Office, London

This time, however, there is not much to complain about. First of all, the line-up for Thursday looks terrific. The expert group is now about 20 members strong, covering a broad range of backgrounds and experiences from all shades of business, government and civil society. Social commerce managers, government innovators, academics, reputation consultants, web developers, social media geeks, consumer spokespeople, top-rated reviewers and the targets of reviews — this promises to be an engaging discussion. Furthermore, we are very grateful for the opportunity to meet in a wonderful venue in the heart of London, thanks to the generous support of Which?, the consumers’ association.

Finally, four members of the expert group volunteered to kick us off with short presentations, highlighting different perspectives on online reviews and ratings:

  • Jason Smith, Client Partner at Bazaarvoice, is a long-time expert on consumer feedback in social commerce: “Reviews and Social Commerce: Learnings from 1000 Brands”.
  • Peter Durward Harris is a Top-10 Amazon Reviewer and will share some of his stories: “My experience as a Top 10 Amazon Reviewer”.
  • Chris Emmins, Co-founder of Kwikchex, will highlight the consequences of public evaluations for individuals and businesses: “Online Reviews – The Good, the Bad and the Downright Ugly”.
  • John Robinson, User-generated Content Lead at NHS Choices, will share his experience with user reviews in the public sector: “Introducing patient feedback on NHS Choices: the challenges and what we’ve learned”.

Feel free to have a look at the preliminary workshop programme. We will make sure to take a lot of notes and post a summary soon after the event.

{ 0 comments }

Share your online review and rating story!

by Admin on March 11, 2011

Do you have an interesting story to tell about online reviews and ratings? Have you ever been troubled by online feedback — or learned something very valuable from using it? What exactly happened and how did you deal with it? Do you have any ideas or thoughts about what could be improved? Have online reviews and ratings changed the way you go about doing things?

Whether you browsed book reviews on Amazon, shared Friday night’s restaurant experience on Qype, have a B&B discussed on Tripadvisor, commented on your tutor at RateMyProfessors, run a garage service reviewed on Google Places or offer reviews and ratings on your own website — please share your experience in the comment section. We are currently collecting stories for the project and would like to find out what you think. Difficult situations, anecdotes, problems, success stories, whatever you think relevant.

We will use these stories in the upcoming expert workshops in London so there is a good chance that your story will be read by people who care about them. Many thanks!

{ 3 comments }