Wine Scoring & Marketing Measurement

February 16, 2012 By Rick Shea Leave a Comment

Tuesday evening I was browsing the latest edition of the LCBO’s ‘Vintages Release Catalogue’. This catalogue provides descriptions and sometimes wine critics’ quality scores for the new wine products about to be released through Vintages stores in Ontario. As I browsed, two thoughts came to mind.

Firstly, I noticed that most of the scores in this catalogue were between 88 and 92 on a 100-point scale. It struck me that this suggested the majority of the wines in this catalogue were of very similarly high quality, with almost all wines rated within a narrow 5-percentage point range. I found that odd, perhaps unrealistic, and decided to think about it. Secondly, I noticed that the wine descriptions were making me thirsty.

Wine Glass, Red Wine

Seeing the wisdom in choosing the beverage that best suited the task at hand, I poured myself a glass of red to compliment my thinking, sat down with the catalogue and made a few calculations and notes. Here are some highlights.

  • Vintages published scores for 57 of the 120 wines in this catalogue. The wine critics quoted used the 100-point scale for 48 of the 57 rated wines. The other nine were based on 20, 5 or 3-point scales.
  • Of the 48 using the 100-point scale, 41 (85.4%) received a score between 88 and 92, and 30 of those were either 90 or 91, which confirmed my first observation. The other 7 wines were rated higher than 92, leaving no scores below 88.

Taking a sip from my glass, I contemplated why so many wines received such similar scores, and how all this relates to marketing measurement. Here are a few thoughts:

Wine Scoring: The LCBO is in the business of selling wine and I suspect they have a policy of only publishing scores of 88 or higher. I tested this theory by looking at the two previous Release Catalogues and wasn’t able to find a wine scoring 87 or lower. Perhaps they’ve learned that lower scores reduce sales and so don’t publish scores below 88.

Marketing Measurement: Marketers are in the business of spending money effectively to drive positive business outcomes. Instead of measuring only the best marketing programs or those you might want to cast in a favourable light, measure and rank all programs so you can identify which are most and least effective, and then optimize future strategies accordingly.

Wine Scoring: By my rough count, the 48 scores using the 100-point scale were sourced from 26 different wine critics. While each used a 100-point scale, I have a hard time believing all 26 used the scale in exactly the same way. I also suspect that some critics are more generous with their scores than others, like my calculus teacher in CEGEP. The other important issue is that scoring wine is a highly subjective exercise. It isn’t at all uncommon for two or more tasters to disagree on a wine’s quality and the corresponding score. Experts have different opinions on subjective matters.

Marketing Measurement: To minimize inconsistencies, reduce or eliminate subjectivity and personal bias from your measurement processes. Having 26 experts using similar but sometimes different methods of scoring your marketing programs based on their personal opinions would not be a recipe for consistency. One person needs to lead your measurement efforts using one methodology that your organization understands and supports.

Wine Scoring: I did a little reading on wine scoring and discovered that wine critics can be inconsistent in the scores they award to the exact same wine on different occasions. For example, the influential wine critic Robert M. Parker has apparently pointed out that he sometimes assigns different scores to the same wine at different tastings, but that those scores tend to be no more than 3 points apart. It seems that differences in tasting conditions and the taster’s emotions can lead to different scores. To address this, I believe Robert Parker publishes average scores when multiple tastings produce different scores.

Marketing Measurement: Consistency is important in making comparisons meaningful. Pick one methodology that can be used consistently across all programs. Consistency should help you to avoid having all your scores cluster within a narrow range where differences may not be significant, or actionable. Programs can differ significantly in their effectiveness at meeting your objectives, and so their scores should reflect those differences. Also, if data for one metric is collected at various times, or from different sources, you might want to follow Robert Parker’s lead and use an average score for that metric.

Advice for Wine Drinkers: Don’t worry about the difference in quality between a wine that scores 88 and another that scores 92.  Both are high quality wines and the difference in scores may come down to who tasted it, under what conditions and the tasters’ preferences. Here’s the fun part. Through trial and error, you should eventually be able to determine which wine critic your tastes best align with, and then the ratings and tasting notes from that critic will help you to make better wine purchasing decisions.

Advice for Marketers: Similarly, there will be some trial and error involved, but not nearly as much fun. Select a measurement methodology that you can apply fairly, without personal bias, and consistently across all programs. Be disciplined about measurement and it will ultimately highlight which marketing programs best meet your objectives and create value for your business. That will help you to make better marketing decisions, which you may wish to celebrate by opening a bottle of your favourite wine!

Filed Under: Approach, Objectivity Tagged With: bias, comparison, Effectiveness, Management, marketing, Measure, measurement

Separating Fact from Opinion

October 20, 2011 By Rick Shea Leave a Comment

Megan Kalmoe rowed for the United States at the Beijing Olympics and is a 2011 World Silver Medalist. While she’s obviously a great rower, I hadn’t heard of Megan before reading Randy Starkman’s article in the Sports section of the Toronto Star on October 13, 2011.

The story focused on the fact that Megan named two Canadian men to the 2011 edition of her “20 Hottest Male Rowers” list, which she recently published on her blog. Not surprisingly, I didn’t make the list. One reason is that I’ve never rowed in my life. There might be other reasons, perhaps many, but more on that later.

I’m happy when Canadians do well in any international ranking, but being a marketing measurement guy, my interest in this article was to learn about Megan’s rating and ranking process.

Here’s what I learned:

  • You must be a world class male rower
  • You may lobby Megan to get on her list
  • Megan’s female friends can nominate you and lobby on your behalf
  • If Megan thinks you’re hot, she might add you to the list

As far as I can tell, Megan’s approach is pretty subjective and unstructured, and the resulting list reflects her opinion, which is fine.  I can’t tell whether lobbying influences Megan’s decisions, but it really doesn’t matter.

It is worth noting that Megan is just having fun with this and important strategic decisions aren’t being made because of her list. But, consider this. What would happen if the top three rowers on Megan’s list landed endorsement deals because they topped her list?

I think the fun and friendly lobbying might get a little more intense for the 2012 list, and Megan might feel the need for a more structured approach to minimize the impact of bias and personal opinion on the rankings.

For now, since this is all in good fun, Megan’s methodology for measuring male rower hotness is perfectly appropriate and as good an approach as any. However, when it comes to measuring marketing program hotness, marketers need a more rigorous approach.

What if you used a similarly unstructured method for measuring marketing programs? Brand managers would lobby you to have their programs highly rated. People working in sales, finance, customer service or operations would also offer their opinions. You wouldn’t have much fact-based data and you’d end up having to make an opinion-based judgment call.

The impact on your judgment call of everyone lobbying to influence your opinion might come down to:

  • The clout of each person doing the lobbying, perhaps related to their role in the organization
  • Each lobbyist’s communication skills and powers of persuasion
  • Your ability to separate fact from opinion, and to somehow remain objective

Here’s the problem. You don’t want the most effective lobbyists to skew the rankings in their favour.  Nor do you want personal opinion and bias tainting your overall approach. Opinions are interesting, but not very actionable.

To be able to take action, make good decisions and adjust strategies, you need data to identify your most and least effective programs. A structured and disciplined methodology will give you that data, while filtering out opinion.

To remove as much personal bias and opinion as possible from your marketing measurement efforts:

  • Involve the Right People: Create a cross-functional group to pick an approach that balances everyone’s needs and interests, so the approach is fair and equitable for all.
  • Involve Unbiased People: People with no vested interest in which programs get the highest rankings could include an analyst, someone from Accounting, or an independent consultant.
  • Set Clear Evaluation Criteria: Disclose how you will consistently evaluate each program, so everyone knows and plays by the rules.
  • Set Objectives Up Front: This prevents people from later setting lower objectives than they would have up front, thus making both their successes and failures look better.

Whether you need to identify your 20 hottest marketing programs, or which types of marketing spending should be increased or decreased, make sure your measurement methodology gives you the unbiased and opinion-free data and facts you need to make better decisions.

As for the fact I didn’t make Megan’s list, I think if I was at least 20 years younger, 10 pounds lighter, 5 times as athletic, a lot hotter and a world class rower, I could have been a contender. Of course, Megan might have had a different opinion!

Filed Under: Approach, Effectiveness, Objectivity Tagged With: bias, Effectiveness, marketing, Measure, measurement, objective

Don’t Ask Me

December 16, 2010 By Rick Shea Leave a Comment

I didn’t do anything wrong, shady or inappropriate, but my final grade in Calculus II was probably higher than it should have been. In the mid-70s, Calculus II was part of my CEGEP studies at Champlain College on Montreal’s south shore.

After a disastrous mid-term exam, I needed either a big comeback or a small miracle to salvage a decent final grade.  While licking my mid-term exam wounds, I resolved to play less road hockey and maybe even study a little.  Then, right after the mid-term, the mother of all academic miracles arrived in the form of a teachers’ strike.

I played a lot of road hockey during the strike, and occasionally I studied.  The strike was eventually settled during the last week of the semester.  I remember walking nervously back into the last class to see what would happen.  I knew my not so vast knowledge of calculus was likely less vast than it had been at the mid-term, and I was worried that the mid-term results would stand as my grade.

Then, it happened. Our teacher explained that there was no point in giving us a final exam, since we hadn’t covered any new material following the mid-term.  Instead, he asked us to pull out a single sheet of paper, write down what grade we each wanted for the course and explain why we deserved it.

All that fresh air and exercise from playing Canada’s national sport had prepared me well for my Calculus II final exam.  With my brain uncluttered by calculus theory, I was thinking clearly and presented my case.

I explained how in recent years I had tended to start slowly in my math courses, with poor mid-terms always followed by sensational final exams, which was true, even if “sensational” might have overstated the case slightly.  I calculated my average grade in my recent math courses and asked for a grade of 70%, which was slightly lower than my recent average.  My teacher accepted my self-evaluation and I got my 70%.

What lesson can we learn from this? That writing skills are more important than math skills?  That playing road hockey is a great way to prepare for an exam?  Actually, I was thinking more about the wisdom of letting self-interested people grade themselves and how this applies to marketing measurement.

When a company wants to measure its marketing, should we ask the marketers to grade themselves? Perhaps no more than a Calculus II teacher should ask his students to grade themselves.  Although I still feel my logic was sound, I’m pretty sure I was biased in creating my scoring process at the end of the course, for the purpose of justifying the grade I wanted.

Here are a few tips to minimize bias in your marketing measurement:

  • Decide on a measurement process before you do your marketing. Before you do any more marketing, if you don’t have a measurement process, or don’t like the one you have, start by creating a new process.  We all know that a fair employee performance evaluation process involves starting the year with clear goals and having a method for scoring results against those goals so you can fairly award merit increases and bonuses.  Treat marketing measurement the same way so everyone knows marketing’s goals and how the results will be evaluated.
  • Involve key functional areas in creating your measurement process. Since one of marketing’s underlying purposes is to improve the overall performance and health of your company, the measurement process needs input from all key functional areas.  For example, suppose the objective of a specific program is to influence customers to order earlier ahead of your busy period, which will mean you can staff more appropriately, operate more cost effectively and manufacture more efficiently.  Then the departments affected need to be part of creating the measurement process and identifying the right performance metrics and objectives.
  • Ask the marketers, but remember their natural bias. If I just spent $100,000 on a marketing program, and the CEO asks me if it worked, I’m going to try to find a way to prove that I spent the $100,000 very wisely.  I can’t help it.  I’ll want to prove that it worked, as opposed to assess whether it worked. It’s in our human nature to have a positive bias towards our own performance, especially when trying to survive and succeed in a dog eat dog corporate environment.  Still, marketers must play a key role in developing an unbiased measurement process.
  • If you’re the marketer, why not take the lead and ask others. By taking the lead, you can show your willingness to be held accountable, and by creating a process up front, you can avoid much of the bias that comes from creating a measurement process after the fact.  Inviting others in the organization into the process further eliminates bias by making sure that interests other than yours are being taken into consideration.  That will also help keep you out of an awkward conflict of interest position, and any perception that you might be biased.

I caught a break back in CEGEP and basically got to pick my final grade.  It was a bit like going in front of the judge, pleading not guilty, and hearing the judge say, “OK, that’s good enough for me!”.  If as my CEO you want to know how well I’m performing as your VP Marketing, my advice to you is to remember that I have a natural bias towards giving myself a good grade, so don’t ask me!

Filed Under: Objectivity Tagged With: bias, Effectiveness, marketing, measurement, objective