After today's meeting, I was reminded of a behavior I came across at my last job that represents an aspect of reputation ratings we didn't discuss.
I worked for a small corporate services firm which relied heavily on communication with in-industry sources (over 23,000 in the db). No one wanted to waste their time calling unhelpful people, so we implemented a commenting/rating system, with a 1-5 scale or helpfulness, expertise, friendliness, etc. The system was a complete failure, but not because people tended gave bad ratings to unfavorable sources, as we discussed in class. It was actually the opposite.
The few ratings that were filled out were almost invariably 5s, the highest rating, but less than 1% of those contacted received ratings. I never saw a score below 4.
When I questioned those using the system about this peculiar use pattern, it became apparent that no one wanted to rate anyone as a 3 or below, because the implication that we dealt with such low caliber sources challenged our position as a top-tier firm. Those who did rate people seemed to do so as a boost to their own pride, that is, when talking to a source who seemed more intelligent or useful than they perceived themselves to be, that source was rated as a 5 in all categories. This source's 'coolness' was then made known to the entire firm, but in a manner that seemed less like information sharing and more like an advertisement that the rater was astute enough to recognize a genius. The result was a completely useless frame of reference.
Sorry to go on so long about this, but I wanted to provide an example of ratings of reputation being used primarily as self-endorsement by the rater, to enhance his own reputation.
So, now we have pride and fury as ulterior motives of reputation ratings... has anyone come across other motives for posting comments and ratings that disrupt the integrity of such a system?
Regardless of all that, I found a paper from the Proceedings of the 3rd ACM conference on Electronic Commerce, Computing and using reputations for internet ratings, that highlights the complexity of determining useful ratings. It gets technical in a hurry, but also offers a summary of research on internet ratings and a good bibliography.
Posted by patrick at February 4, 2003 11:34 PMThere has been some good work on recommendation ratings, mostly from some of the original Collaborate Filtering group at UMinn. One, Ben Schafer knows his stuff: http://www.cs.uni.edu/~schafer/research.htm
Posted by: donturn on February 5, 2003 01:53 AMWith regard to Patrick's post about ratings to boost the rater. This anecdote was on my mind in class as we spoke about rating systems with regard to Social Networks:
I'm sure most of you are familiar with buyer and seller ratings on amazon.com. Recently, my husband and I came across a seller who turned out to be how-shall-I-say..."unethical". After he was several days late delivering the book my husband ordered, my husband contacted him. After several emails a week later, my husband rated this seller with less than a glowing score. The seller in turn gave my husband a bad rating - even though my husband has never sold an item. Basically:
Amazon also has "buyer ratings" where, even if you have never sold anything, someone can provide comments based on absolutely nothing. So if you ever intend to sell something, this could hurt your potential.
There always seem to be grievers who are going to find unethical workarounds that work in their favor before there are fixes in the code that go live. That said, I'm a big fan of the workaround as a means to feed product development. Often it is the users, in parallel with dev teams that come up with the most valuable non-version-1.0 features on their own.
Amanda
Posted by: Amanda on February 5, 2003 03:46 PM