10.11.2010

Good Food Award judging

As mentioned earlier, I was lucky enough to be selected as one of the judges for the coffee track of the Good Food Awards.

I was honored and flattered by my inclusion - and a bit intimidated by the list of other judges. As a result, I went into the experience looking to learn as much as I could from the opportunity.

In the end - that was a good thing, as there was an enormous amount in the process and experience for me to learn from.

A day later - with marginal time to think and digest I admit - it feels like there are a couple really big take-aways (for me) from this amazing experience.

Taste truly is subjective

I have, for years now, told everyone that taste is subjective and that universal pronouncements about taste are pointless and incorrect and perhaps destructive.

But this event brought home not only how true this really is - but also how little I'd actually accepted the truth of the statement.

It was incredible to see a single coffee get scored 90+ points by one judge and 80 or below by another -- and to have each judge present a compelling, rational and well-reasoned argument for why their scoring was correct. It's amazing to realize that both judges can, in fact, be correct -- despite this enormous disparity in judged value.

It was even more incredible to see these judges make their arguments - while respecting the differing opinions.

This made me realize that while taste is (really and truly) subjective - opinion is not of equal value. Someone presenting a clear and complete and logical argument for why something I hate is actually good results in my respecting their opinion and the truth in their judgement (while not changing my own opinion at all). On the other hand, someone agreeing with me but being unable to give any clear and enunciated explanation for why has little to no validity.

The so-called "under-development epidemic" might be overstated

Development (full, under, "proper") was definitely a topic of discussion - and was a source of enormous controversy at times. But the number of coffees that were severely under-developed (in my opinion) was actually quite small. Out of 51 coffees, there were probably less than 10 that were really under-developed and fewer than 5 were severely so.

Now... what was most interesting is that there was only once coffee (to my recollection) that was under-developed and was an average or inferior green coffee. The under-developed coffees were more likely to be very (very) high quality green. This is (obviously) a concern. But it also points to a possible conclusion - that much of the under-development of these coffees is due to fear or timidity when roasting.

That being said - the controversy around under-development was very real - and there were judges on both sides of the argument.

Technique is objective, style is subjective

Perhaps the single largest challenge of judging this event was figuring out how to approach the stylistic differences issues.

There are styles of coffee that each of us prefer - and styles that each of us doesn't like. I (for example) really don't like dark roast indonesian coffees. That is a stylistic preference.

The problem is that there are those who love that style. So how does one judge a style that you simply don't prefer?

In the end - I had no choice but to try and judge against the style rather than just my preference. In other words, I tried to look at the coffees and evaluate the technique behind the coffee rather than the style of the coffee.

So a well-executed dark roast east african natural (while not to my own taste) would score well if the technique were ideal and the beans well selected (again for that style).

The idea is that the technique of roasting (the craft) should be universal and tangible - while the style of the coffee can be personal and subjective.

This is not the way any of us have been trained to cup - and it's a hard shift to make.

The largest problem in US coffee is old green

It was shocking to me (and I think to many of the judges) to encounter so much past-crop or baggy or just old green coffee. I think I probably cupped more baggy coffee yesterday than I've cupped in the last 2 or 3 years.

The old coffee issue was (for me at least) more of a problem than roast degree and more of a problem than dirty coffees and more of a problem than any stylistic differences.

If I were going to give one single piece of feedback to roasters in the US based on this event it would be, "stop buying / selling old coffee." I don't know if the issue is that roasters are buying more than they can sell before the coffee goes off - or if the roasters are buying old baggy coffee from the importers. But it doesn't really matter in the end. What matters is that the consumer is being sold inferior coffees.

If roasters want to move the needle the most with the least effort - this is where they should focus. Stop selling old coffee.


Finally, a couple side notes.

1 - The judges were amazing. Honestly, it was amazing to cup and have these conversations with such incredible people. I am not worthy. I was probably the least experienced cupper of all the judges. But I was welcomed by all. It was great. Thank you.

2 - I have to give huge props to the folks responsible for this event. To the organizing committee... to the volunteers... you all rock. And big thanks to Peter Guiliano for your fantastic facilitation efforts and to the crew who worked all day Saturday to select the top 50 for us to judge. Most of all, it absolutely could not have happened without the incredible hard work of Brent and Mie. Thank you too so much.

4 comments:

jeremybb said...

Chris,

You may remember as the beer judge who attended your monster cupping a couple months ago. The points you make about style vs. preference are something that comes up all the time in beer judging and since I know you are also a beer fan, you might be interested to see how beer judges approach this.

Beer judging is meant to be--just as you say--all about adherence to style, rather than hedonics (I guess the presumption is the styles are good, at least to someone). Here is maybe where I disagree with one thing you said: Technique *and style* can be objective; it's interpretation/perception of adherence to style that is subjective. Beer judges use very detailed, even quantitative style descriptions in their judging. This is good as it makes judging somewhat more objective, and makes the results a bit more intelligible. On the bad side, it is hard to accommodate and reward creativity in such a format, as there will not be a defined style for a new concept in product.

I could say a lot more but since this is coffee not beer I'll leave it there, but you can find all thing beer judging at:
http://www.bjcp.org . The most relevant links to this discussion are here: http://www.bjcp.org/examcenter.php

JennyReb said...

Thanks for your insights from the GFA Awards, Chris. I hear you on the Old Coffee problem, and although I agree with you, I am not sure there is an easy solution for any size roaster to get around this problem. The process of buying just the right amount of green beans (not including direct trade) so that you don't end up with old product is one of the most difficult things to master in the roasting business. The booking forward process is such that a roaster must be able to accurately forecast what they will need for the next 6 months to a year into the future (or basically, until the next season's beans start shipping) in order to obtain the beans they need, because many of the really high quality beans will sell out - sometimes before they even start shipping. Even then, during that 6-month to year period, a roaster's business can greatly fluctuate, so those booked forward numbers can, unfortunately, end up being really far off from the roaster's original prediction. In general, a roaster (speaking for myself only) would probably rather end up with more beans than they need, than too few, because the latter is much more work to compensate for in mid-season if you're trying to replace a crucial blend component in a 5 to 6-bean signature espresso blend for example. It's therefore, very easy for a roaster to end up with too many "old" beans. I can't speak for brokers, but I imagine that it's a similar problem, with prediction of the future market for each of these coffees. Buying green beans in 132lb increments at $500 a pop is not an easy amount for a roaster to just swallow and decide to throw away, simply based on the fact that the product is "old", especially when the majority of consumers will likely not be able to tell the difference.
And buying "spot" coffees is often not a better solution either, because these are usually the coffees that are left over from what roasters didn't book at the beginning of the season or lower quality than what you might be looking for.

So yeah... I don't know.

If it were up to me, I would rather be able to buy beans in slightly smaller increments, like 50lbs. But I'm not sure that's such a great solution either.

~Jen St. (Scarlet City)

JennyReb said...

btw, I don't mean to presume that you, personally, aren't already aware of how the bean buying process works and the dilemmas involved. I just thought it'd be helpful for anyone else reading this who may not.

~Jen St. (Scarlet City)

chris said...

I think learning from the way judging is done with other food/beverage products would be hugely valuable.

The coffee world doesn't really have great methods for judging final product across multiple styles, roasters etc. It's not something we've addressed well.

Then again, if there is one thing that the GFA experience taught me, it's that the problems that come from an event like this are rarely going to be judging methodology related. We could fix all the processes for evaluating coffee; we could optimize perfectly and the end results would be no better.

My belief at this point is that a true evaluation of great coffees and roasters on a national or global scale will not be possible within the structure of an event and organization like the GFA. Without an understanding of the realities of coffee at the highest level of the organization, the results will be at best compromised (and at worst entirely useless and pointless).