Jump to content
Gwydion Stone

Using WS Absinthe Review Module

Recommended Posts

Close those parentheses ya Sixer lookin' bastid. ;)

 

Regarding FV vs WS, I've had about equal amounts of people comment on both sides. I like the FV system too, but I find it entirely too open ended, for the reasons mentioned above.

Share this post


Link to post
Share on other sites

7 point system!

 

I see it both ways, actually. The 5 star system keeps things simple but perhaps a bit too simple. The 10 system might be too open. This is all subjective. I just somehow find it hard to be completely objective with any rating system. I'll stop speaking in truisms here until I have an actual suggestion to contribute...

 

Thinking...

Share this post


Link to post
Share on other sites
Only if you think it needs a 4. Just look back through the reviews of other high rated absinthes and you'll find that many were given a 5 in the overall category.

Some could have been mistakenly done because the reviewer might not have understood the definition of a 5. But more often, those 5s were given before the criteria was changed to make it harder to score an absinthe an overall 5. I notice I still have a couple I haven't changed to reflect the new definition. I'll be doing that today.

If the board is concerned about an un-biased review database, I wouldn't think it's in their best interest to change the criteria and make scoring tougher after there own brand(s) have been released and reviewed using the old system and then keeping them side by side with brands reviewed off of a new, tougher system. Especially when there's nothing saying that such things are going on. That could be seen as dishonest/biased.

 

In the end, I think if someone is going to make a connection and say the review database is biased towards members, there's not much that could be done about it (I suspect it's going to happen no matter what happens). The members here that have gone commercial make fine products. They're passionate about the drink, they're not just someone who's trying to cash in on the newest thing. It shouldn't come as a surprise that their products are outstanding.

 

I also understand that we are dealing with a 5 point system, and that a prefect "5" score, at least compared to 100 point wine scoring systems, is the equivalent of a score of 90 - 100.

Wouldn't it be 80-100? What percentage of those "approximately 205,275" brands scored 80+?

Share this post


Link to post
Share on other sites
I wouldn't think it's in their best interest to change the criteria and make scoring tougher after there own brand(s) have been released and reviewed using the old system and then keeping them side by side with brands reviewed off of a new, tougher system. Especially when there's nothing saying that such things are going on. That could be seen as dishonest/biased.

Well if you put it that way...

 

The criteria was changed before many of the WS brands came out, at least according to my records. The ones that I had previously given a 5 overall were not WS brands. They included Coquette, L'Artisanale, Bell Amie and a few others I don't remember. I think the only one that had been out at that point was Swiss Marteau, which didn't score a 5 anyway.

 

And there were several mentions that the criteria had changed, as far as I know.

 

As I mentioned before:

If you truly think that a brand cannot be improved upon, then 5 it is. It's your opinion, and it's perfectly valid and will be respected.

Share this post


Link to post
Share on other sites

My mistake. I figured the change was recent as I try to keep up with the absinthe world as best I can and didn't remember ever reading about it. I still stand behind the second part of what I said (which would include those same brand(s)).

Share this post


Link to post
Share on other sites
In the end, I think if someone is going to make a connection and say the review database is biased towards members, there's not much that could be done about it (I suspect it's going to happen no matter what happens).

Exactly--we can strive to keep the review section in as tip-top shape as we can, which includes having these sorts of discussions. If we keep our side of the street clean--which of course is an ongoing process--then if they perceive such things it's their issue because we know we've done the best we're able.

 

Precenphix thought he had cornered the market on truisms! :devil:

Share this post


Link to post
Share on other sites

Just trying to keep the peace and put things in perspective...and failing miserably on the later. :blush: I tend to blab a bit.

Share this post


Link to post
Share on other sites

Oh! Sorry, I didn't mean it that way--I thought I was overusing truisms, so I perceived myself wresting that mantle from your paws. I mean headphones.

 

On topic--I've thought adding a .5 to the numerical rating system here, though that could make things more complicated. And I think it's already been covered. :blush:

Share this post


Link to post
Share on other sites
I also understand that we are dealing with a 5 point system, and that a prefect "5" score, at least compared to 100 point wine scoring systems, is the equivalent of a score of 90 - 100.

Wouldn't it be 80-100? What percentage of those "approximately 205,275" brands scored 80+?

 

No, it actually is 90 - 100 because most of the 100 point wine scoring systems are based on Robert Parker's system which is a system of 50 objective points stacked on top of a baseline of 50 points. A wine that received 0 points in each of Parker's four defined categories would have a score of 50 points. In other words, if a wine attains 80% of the objective points allowed by the system, that is a 90 point score.

 

I was happy when I discovered, the other day, that some descriptions of scoring within the categories on the WS evaluation sheet had been changed. I was especially happy that the description for a score of "5" in "Overall" had been better defined, since the former description "I'll have another, thank you!", was no objective criteria, and essentially useless.

 

Soon, I had planned to revisit each of my reviews, in part to make decisions about which ones were deserving of a score of "5" overall. Now I'm not sure what to do. It is my opinion that a "5" rating overall should be allowable if the reviewer judges the product to be in the very top echelon, not just when he judges it to be impossible to improve. I'm not sure how many people here are capable of making that judgement, certainly not me.

 

Initially, I saw the new description as very useful, but limiting. There is a difference between "Perfect. No improvements needed." and Perfect. No improvements possible.

Share this post


Link to post
Share on other sites

Thanks for the reply on the wine rating thing. I had no idea that the lowest score a wine could get was a 50.

 

It is my opinion that a "5" rating overall should be allowable if the reviewer judges the product to be in the very top echelon, not just when he judges it to be impossible to improve.

I'd have to agree.

Share this post


Link to post
Share on other sites

I'd think that in a five star rating system you certainly should be able to use the fifth star, otherwise you just have a four star rating system. :fork:

 

Restaurants and hotels all have five star rating systems, and there are certainly establishments that receive all five stars.

Share this post


Link to post
Share on other sites
since the former description "I'll have another, thank you!", was no objective criteria, and essentially useless.
That's exactly why we changed it.
It is my opinion that a "5" rating overall should be allowable if the reviewer judges the product to be in the very top echelon, not just when he judges it to be impossible to improve.

We never said that we will prevent someone from using a 5 overall score. What I said was, use it at your discretion, but with trepidation and the knowledge that every 'really good' absinthe that comes out shouldn't be getting a 5. Just like every really good wine isn't getting a 100 score.

The problem there lies in future releases. Let's say a brand scores an overall 5 now. Speaking from experience over the past 10 years, production of absinthe will continue to improve. Does that mean, 4 or 5 years from now EVERY new absinthe that comes out will score an overall 5? When DPs product continues to be refined and she has the ability to age product, I'm sure it will get better (although it's great now). Scoring a 5 now means when the improved product comes out, it will score the same, and thus appear that it wasn't improved upon.

 

Initially, I saw the new description as very useful, but limiting. There is a difference between "Perfect. No improvements needed." and Perfect. No improvements possible.
So do we change it, or keep it how it is?

Share this post


Link to post
Share on other sites
I'd think that in a five star rating system you certainly should be able to use the fifth star, otherwise you just have a four star rating system. :fork:

 

Restaurants and hotels all have five star rating systems, and there are certainly establishments that receive all five stars.

But VERY VERY few do. And those scores are also reviewed yearly if not more. I don't see most reviewers on the page to have enough interest in it to go back and rescore their ratings every year.

 

As I mentioned before:
If you truly think that a brand cannot be improved upon, then 5 it is. It's your opinion, and it's perfectly valid and will be respected.

Share this post


Link to post
Share on other sites
Restaurants and hotels all have five star rating systems, and there are certainly establishments that receive all five stars.

 

 

Except for those Diners, Drive-Ins, and Dives that use the Michelin three star rating system, of course. ;)

Share this post


Link to post
Share on other sites

Which also is reviewed regularly and stars either given or taken away. For comparison's sake, roughly 0.0005% of restaurants in the world receive 3 Michelin stars, the equivalent of 5's across the board for a WS review.

 

0.28% of hotels and restaurants receive the coveted 5 Diamond award from AAA.

 

For some interesting insight from a wine guy about ratings, take a read here.

 

In my mind, giving all 5s for the qualities, then an overall of 4 gives it the equivalent of a Robert Parker wine score of 90-99, which he defines as:

equivalent to an A and is given only for an outstanding or special effort. Wines in this category are the very best produced of their type. There is a big difference between a 90 and 99, but both are top marks. As you will note through the text, there are few wines that actually make it into this top category because there are not many great wines.

 

A 5 in overall, would be the equivalent of giving Robert's wine a 100, which is almost as rare as the 3 Michelin star/5 Diamond award. Wine Spectator listed its Top 100 wines of 2008. Only 1 received a score of 100. Only 1 in 2007 and NONE in 2006.

Share this post


Link to post
Share on other sites

Ok, I'm still confused. Brian, you said ""If you truly think that a brand cannot be improved upon, then 5 it is." But as fpb pointed out, there is a difference between "no improvement needed" and "no improvement possible". The overall category in the WS review says the former, while your comment (cannot be improved upon) seems to imply the latter.

 

When I first read your comment, I thought "ok, that means if any category gets less than a 5, that really indicates that the absinthe can be improved, so the overall cannot be a 5". So I revised my MoL review, and changed the overall to a 4, and I had given a 4 to the aroma. But now, taking into consideration that the wording is "no improvement needed", I really feel that it can be a overall 5. (I'm going to let everything chill for a bit, give my rapidly eroding credibility a well-deserved rest, and maybe re-review if I have time over the weekend)

 

I really like fpb's suggestion of "very top echelon", but would that be considered too subjective? I'm actually very comfortable with such a notion; many years ago, when I was first getting into single malts, I had 5 groups that I'd lump them into after tasting. The ones that I had in my top group (my "very top echelon") were considered roughly of equal quality. I never considered whether or not they could possibly be improved, just that they were the best of the best. There weren't many that I put into this bucket (my poor memory tells me that the distribution was bell curve-ish), and occasionally, as I got more malts under my belt, some would drop out.

 

All my rambling aside:

  • When it gets to this top echelon, I'm not qualified (and probably never will be) to judge whether or not something CAN be improved. If that's the criterion for an overall 5, I can't make that judgement.
  • If the intent is really "no improvement needed" vs. "no improvement possible", we may want to update the description, as the distinction is subtle and was certainly lost on me. "Very top echelon" (or "in an elite class") actually suits me fine, but may be considered to be too subjective.

Share this post


Link to post
Share on other sites
Ok, I'm still confused. Brian, you said ""If you truly think that a brand cannot be improved upon, then 5 it is." But as fpb pointed out, there is a difference between "no improvement needed" and "no improvement possible". The overall category in the WS review says the former, while your comment (cannot be improved upon) seems to imply the latter.

This is the problem every review system gets into. Arguing about symantics.

 

I think I've explained pretty well what my thoughts are regarding the definition of a 5 overall score.

 

I'd like to get as many people as possible to weigh in on it. If the former definition is preferred, then that's what we'll go with. If the latter is preferred, same deal. I just want to make sure it's UNIFORM and that everyone understands the criteria.

 

Shall I set up a poll? Would that be the easiest way to get as many votes as possible in the shortest amount of time?

Share this post


Link to post
Share on other sites

"No improvement needed" sounds more like a four, to me (or somewhere between a four and a five, although I realize there's no room for such a breakdown under this scoring system), while "no improvement possible" has the ring of a five. Keeping in mind that it's still one's subjective idea of what couldn't be possible, but, in fact, might be.

Share this post


Link to post
Share on other sites

Do the powers that be want an absinthe review database made from only review experts, or by anyone who wishes to submit a review? If it's the former, then reviews should only be allowed to be submitted by a select few. If it's the latter, then the powers that be are going to have to deal with some reviews being filled out in ways in which they may not like or agree with. I don't think you can have it both ways.

Share this post


Link to post
Share on other sites

I don't think there is any question, WS wants as many reviews as possible. I also don't think there's any question, we'll discuss what the numbers are suppose to mean, again and again. It is difficult to make subjective opinions objective but ultimately, that is the goal.

 

I've also wanted a .5 from time-to-time, myself. If it was difficult deciding between a 3 or a 4, a 4 or a 5, a .5 would have been a near-godsend.

 

But I'm not advocating the change, due in large because all the reviews to this point have been made without it. I'd hate to skew and screw-up all the work that's already been done.

 

This has been a good conversation. I think it's time for me to re-visit my reviews. :cheers:

Share this post


Link to post
Share on other sites
If it's the latter, then the powers that be are going to have to deal with some reviews being filled out in ways in which they may not like or agree with. I don't think you can have it both ways.

The latter of course. But that is also why they asked to have a Review Editor.

 

If something looks out of the ordinary, additional information is gathered to see if 1) The review was submitted by a brand rep, 2) The review was submitted with a misunderstanding of criteria, 3) The review was submitted by someone trying to intentionally make a brand look bad, or 4) The review was submitted appropriately.

 

We have tried to make the criteria as easy as possible to submit a correct score without any way of needing to argue about what the numbers mean. Does it look natural? Is the louche appropriate? Does it taste basic, or is it well formed and complex? Is it appropriate for it's classification?

 

 

The numbers should be objective based on the stated criteria. The comments should be subjective.

 

As mentioned ad nauseam, there will always be outliers both on the high end and low end of ratings for a particular brand. The more reviews that are posted, the less impact those outliers will have on the overall score.

Share this post


Link to post
Share on other sites
It is my opinion that a "5" rating overall should be allowable if the reviewer judges the product to be in the very top echelon, not just when he judges it to be impossible to improve.

What I said was, use it at your discretion, but with trepidation and the knowledge that every 'really good' absinthe that comes out shouldn't be getting a 5. Just like every really good wine isn't getting a 100 score.

 

Agreed

 

When DPs product continues to be refined and she has the ability to age product, I'm sure it will get better (although it's great now). Scoring a 5 now means when the improved product comes out, it will score the same, and thus appear that it wasn't improved upon.

 

It is possible that a product is being improved or changed, but within the strata of the very finest of its type. I don't think little tweaks within such a narrow range will ever be accounted for adequately with a 5 point rating system.

 

Initially, I saw the new description as very useful, but limiting. There is a difference between "Perfect. No improvements needed." and Perfect. No improvements possible.
So do we change it, or keep it how it is?

 

Not sure yet, but I think this conversation is helpful to guide us.

 

Which also is reviewed regularly and stars either given or taken away.

 

roughly 0.0005% of restaurants in the world receive 3 Michelin stars, the equivalent of 5's across the board for a WS review.

0.28% of hotels and restaurants receive the coveted 5 Diamond award from AAA.

 

I think we're loosing track of the fact that we're not talking about 5s across the board, we're talking about 1 component of the final score. I agree that 5s across the board is the equivalent of a Parker 100. The final score is the final score. A final score of 4.5, to my way of thinking, would be the equivalent of Parker's 90 points, and there are certianly currently produced absinthes that are deserving of that.

 

Right now, we almost seem to be making the mistake, in my opinion, of treating the category of "Overall" as if it is the final score. The implication is that an overall impression of 5 is only deserving when a product receives all 5s in the other categories. I think it should be treated as the reviewers overall blended impression of the product they are reviewing.

 

A 5 in overall, would be the equivalent of giving Robert's wine a 100...

 

I don't think so, see above.

Share this post


Link to post
Share on other sites
Right now, we almost seem to be making the mistake, in my opinion, of treating the category of "Overall" as if it is the final score. The implication is that an overall impression of 5 is only deserving when a product receives all 5s in the other categories. I think it should be treated as the reviewers overall blended impression of the product they are reviewing.

 

A 5 in overall, would be the equivalent of giving Robert's wine a 100...

 

I don't think so, see above.

But the only time one would even be able to consider giving an overall score of 5 is if all other criteria score a 5.

 

One can score 5's in all areas, but still a 4 in 'Overall', but based on the criteria, there isn't a way were a brand could score less than 5 in all areas, but then a 5 in 'Overall'.

 

Anything less than 5 in any of the any other criteria indicates that there can be an improvement made.

Share this post


Link to post
Share on other sites

So (to use my favorite example) I gave Pacifique a 5 in every category but color, where I lowered it to a 4 because I thought it was just a wee bit on the pale side, and by your reckoning, I'm required to give it a 4 overall?

 

Given that it averaged out to a 4.8 in all the other categories, I think this is too rigid, and doesn't accurately reflect the overall quality of the product...a 4 is tantamount to an 80%, no matter how you slice it, and there is no way that I could justify giving Pacifique that kind of score on the whole...I'd sooner reinstate the original 5 I awarded it for color.

 

The big problem for me (particularly with the overall category) is that there is a tremendous disparity between a rating of 4 and a rating of 5. If I were able to give it a 4.8 overall, that would make far more sense to me. Perhaps the overall category should be broken down further to allow it to serve as a reflector, or average of all the other categories.

 

Just my .02.

 

Rant off, btw. :)

Share this post


Link to post
Share on other sites

If we go by either definition above about what a 5 overall signifies, then yes, if you give a 4 to color, then overall shouldn't be 5, since you readily admit you think the color can be improved upon.

 

You can note in the comments your thoughts about the scoring and its limitation.

Share this post


Link to post
Share on other sites

I think I'll simply take the other option, since I really do find Pacifique's color to be quite appealing. I only quibbled with it as excuse not to give Pacifique a perfect score, however, the longer I enjoy it, the more I realize that it's as perfect for me as it gets.

Share this post


Link to post
Share on other sites
I don't think there is any question, WS wants as many reviews as possible.

 

But I'm not advocating the change, due in large because all the reviews to this point have been made without it. I'd hate to skew and screw-up all the work that's already been done.

 

Agreed.

 

...that is also why they asked to have a Review Editor.

 

If something looks out of the ordinary, additional information is gathered to see if 1) The review was submitted by a brand rep, 2) The review was submitted with a misunderstanding of criteria, 3) The review was submitted by someone trying to intentionally make a brand look bad, or 4) The review was submitted appropriately.

 

And I think this is necessary... And I'm glad it's you, not me, Brian! :dev-cheers:

Share this post


Link to post
Share on other sites
Right now, we almost seem to be making the mistake, in my opinion, of treating the category of "Overall" as if it is the final score. The implication is that an overall impression of 5 is only deserving when a product receives all 5s in the other categories. I think it should be treated as the reviewers overall blended impression of the product they are reviewing.

 

A 5 in overall, would be the equivalent of giving Robert's wine a 100...

 

I don't think so, see above.

But the only time one would even be able to consider giving an overall score of 5 is if all other criteria score a 5.

 

One can score 5's in all areas, but still a 4 in 'Overall', but based on the criteria, there isn't a way were a brand could score less than 5 in all areas, but then a 5 in 'Overall'.

 

Anything less than 5 in any of the any other criteria indicates that there can be an improvement made.

 

Then the definition of what attributes qualify for a rating of "5" overall need to be changed again, since it removes one fifth of one of the building blocks for a final score, except in rare instances.

 

Also, and I quote from the WS review guidelines;

 

"OVERALL IMPRESSION

This is your overall feeling about this absinthe. You can equalize any deficits you feel may have resulted from the scorings above. While an absinthe might technically score lower on the above criteria, it may have some other quality that mitigates that deficit."

 

I have never been comfortable with the idea of "perfection" being the standard for the rating of "5" in any category. Extreme excellence should be the standard.

 

Imagine this;

 

0-----1---------2--------3---------4---------5----P

 

 

That's the way I see it.

 

Brian, it seems you see it;

 

0-----1---------2--------3-----------4-----------5

 

 

And that is what effectively happens when you define a rating as perfection, let alone how you ever get an effective number of people in the user base here to understand "perfection".

 

In this scoring system;

 

1 implies a range.

2 implies a range.

3 implies a range.

4 implies a range.

5 at least at this point is... exactly that. An exact point. How do we know? As long as perfection is the criteria, a rating of "5" in any category is as useless as the former definition for "Overall".

 

5 should also imply a range. It should be narrow, elite, and well defined.

Share this post


Link to post
Share on other sites
If we go by either definition above about what a 5 overall signifies, then yes, if you give a 4 to color, then overall shouldn't be 5, since you readily admit you think the color can be improved upon.

 

You can note in the comments your thoughts about the scoring and its limitation.

 

In my case, my reviews can become even wordier than they already are! :tongue:

 

I think I'll simply take the other option, since I really do find Pacifique's color to be quite appealing. I only quibbled with it as excuse not to give Pacifique a perfect score, however, the longer I enjoy it, the more I realize that it's as perfect for me as it gets.

 

And that is why some range in each rating in each category has to be part of the system.

 

I stated earlier that it was my plan to revisit each of my reviews in July, in part, to decide which were truly deserving of ratings of "5" in overall. While it is not done, my current inclination would be to give a "5" overall to;

 

Jade PF1901

Jade Edouard

MAdlBE

Pacifique

 

and... drum roll......................

 

MOL... Yeah, It blows me away that much! Hip, Contemporary, Creative, Respectful of Tradition, Well Executed, and Very Artistic.

 

Now at first glance, while 5 of the presumably 22 - 24 reviews I will have posted at that time may seem like a high percentage, keep in mind that the absinthes I've reviewed are, for the most part, very top tier offerings, thanx in large part to this site and this forum. If I had the time, money, and inclination to review another 40 offerings of far more pedestrian quality, the number would be right down under the 10% range that I think might comprise the very top echelon, to which I refer, in these posts.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×