I've been thinking about it all day and this is tougher for me than an absinthe definition!
The argument now isn't about what it says. The discussion is about what it SHOULD say.
Just what you've seen from me in this one category was tough enough to concoct, as well. Try rewriting all six categories, which I've done. I've been at it, with almost daily review and tweaking since Brian agreed to consider some changes in this
It makes more sense to me that the first question should be "Is this/will this be useful to the reviewer, in better guiding them to assign the most correct scores possible as they construct their review?" That's who these guidelines are written for. The "review-reader" is not necessarily aware of the guidelines that helped construct the review. If improved guidelines result in better and more accurate reviews, then the consumer is ultimately served.
The reason I approached this by rewriting the scoring criteria for scores of 2 through 5 in the category of "Overall", rather than just the criteria for a score of 5, is because, like Brian, I am also concerned about the possibility of overly high scores. It has been my opinion for some time that one of the reasons for some of the overly high scores here is that the scoring criteria in the previous sheets for all categories is derogatory or strongly critical fully through a score of 3. In fact, on the first sheet I used, there was a level of reservation in the criteria for "Color" fully through a score of 4. That only leaves scores of 4 or 5 in most categories to indicate "acceptable" or a compliment. I think that what happens as a result is that many reviewers, when reviewing something they think is "pretty good" default to the levels of scoring that are not derogatory, because the level that should indicate a solid mid-market offering (3), seems too critical. My approach here is to first define criteria for a level of 3 in each category, and then "write down" to scores of 2 and 1, and "write up" to scores of 4 and 5.
The point I would like to make here is that not just the top level (5) of each category must be well defined, but that all levels (1-5) of each category must be very well defined
. The descriptions for scoring need to be well defined, and speak to what should be expected at each level, in a matter of fact way.
When I rewrote the descriptors for the category of "Overall", I started with this standard, which I have applied to all categories;
Level 1. A completely unacceptable standard.
Level 2. A barely acceptable standard. Highly flawed.
Level 3. An acceptable standard. A mid-market offering, however not without some flaws.
Level 4. An elevated offering. Clearly above average.
Level 5. The very finest. Flawless or nearly flawless.
Of course from there, it becomes a simple task to write the descriptors.
So that is the logic I applied to this
, and to the other categories.
Thinking of WS review guide 10 years down the line, I'm sure we'll be glad the system was changed now rather than later. While it is unfortunate for the other reviews, I dont think they will be viewed as less accurate, just not as "specific".
I have never been for adding levels (or points or half-points) to this system. I would much rather see the current system tweaked to make it better. I think a 5 point system is most useable by a wide spectrum of people, and that is what I think this site is all about. Also, my man J.L. said it best.
Joe Lagate May 14 2009, 09:52 AM
I've also wanted a .5 from time-to-time, myself. If it was difficult deciding between a 3 or a 4, a 4 or a 5, a .5 would have been a near-godsend.
But I'm not advocating the change, due in large because all the reviews to this point have been made without it. I'd hate to skew and screw-up all the work that's already been done.
I think we should take best efforts to maximize the effectiveness of the current 5 point system first before we go turning this whole thing on it's head.
And that's the major issue. This isn't MY decision at all. I just review the reviews. I don't have it within my power to just arbitrarily change the review system on a whim. It's not my place to do so, since the system was developed by Hiram with input from the entire advisory board and more.
I like FPB's new classifications, including the term 'uncommonly', so I'm all for changing it to that.
I'm also open to discussion about whether we need to add half-stars to the categories. My only concern with that will be the description of those halves.
Well, you've heard my opinion. If half-stars prevail, someone else can write it!
So can we get some input on what people's descriptions for the .5 star increments would be?
I'd love to take a stab at it myself, but it will have to be after Tales.
And I only responded now because of this post. I know Tales is the raison d'etre for many here right now. Also, I think this should be carefully considered, since it is so fundamental to the mission of the WS. We should take our time and proceed with all due care.