measurement — Optiv8 Consulting

Measure Well, Sleep Well

March 15, 2012 By Rick Shea Leave a Comment

If you know me or have been reading this newsletter for any length of time, you may know that photography is my favourite pastime. What you may not know is that organizations sometimes bring me in to take photos of their events, which is how I found myself at the AllerGen 2012 Annual Research Conference.

AllerGen is a not-for-profit organization whose role is to mobilize Canadian science to reduce the illness, mortality and socio-economic costs of allergic disease. The conference showcased the latest research in this regard and while often over my head scientifically (not hard to do), I found it quite interesting.

During an afternoon break at the conference, a distinguished looking gentleman named Douglas Barber approached me to talk photography. Our pleasant conversation eventually shifted to the conference and he told me a story that I quickly realized fit my thinking on marketing measurement.

Douglas explained he is on AllerGen’s board and that an issue of concern to him is the cost to the Canadian economy from the “asthma drag” on productivity. He explained how asthmatics can be less productive at work or even miss entire days of work following sleepless nights caused by asthma. Parents of asthmatic children can also experience the same productivity losses. Douglas also told me how he once did a quick “back of the envelope” calculation to estimate that asthma costs our economy between $10 and $20 billion per year in lost productivity.

Sometime after Douglas did his quick calculation, a full study was done to properly analyze and estimate the economic impact of asthma’s drag on productivity. The study concluded the annual costs are $15 billion. That’s right; a costly and complex measurement process produced the same answer as one expert using a pen and the back of an envelope.

Two aspects of this story relate to my views on marketing measurement:

Douglas’s back of the envelope calculation relative to the full study is similar to how a marketing scorecard can be a proxy for a sophisticated and costly marketing measurement process. In both cases, the less sophisticated approach doesn’t need to be perfect, just accurate enough to support analyzing options and making the right decisions. As I like to say, it’s not about precision, it’s about the decision.

The back-of-the-envelope estimate worked because it was done by an expert using a sound methodology. Douglas has an extensive business background and apparently knows more than just a little about productivity and related calculations. Scorecards are a proven methodology that you can enhance with expertise about your marketing and your business.

There is another lesson in Douglas’ story, and that’s the need to right size your measurement efforts to the magnitude of the decisions you need to make.

Research Investment Decision

Douglas’ back of the envelope calculation and the full-blown study produced essentially the same estimate and both pointed toward making the same decision. It’s a pretty compelling proposition if investing perhaps a few hundred million dollars into research would lead to recovering even just 10%, or $1.5 billion of the lost productivity, especially as that benefit would be realized every year.

The problem is that any decision to potentially invest a few hundred million dollars needs to be substantiated by more than a back of the envelope calculation. In this case, the cost of the research needed and the probability of recapturing that 10% are two other variables that I think would need to be estimated. It’s understandable that a full-blown study was needed to examine the overall business case.

Marketing Investment Decision

Similarly, for companies that invest tens of millions annually in marketing, it makes sense to support the decisions that need to be made with sophisticated marketing measurement efforts that might cost hundreds of thousands, or more.

For most companies with smaller marketing budgets, a practical lower cost approach such as one using a scorecard may well be the right sized measurement solution. In most cases, the overall measurement expense likely needs to be a small single digit percentage of the total marketing budget.

I like simple and elegant solutions that deliver what you need. A marketing scorecard’s simplicity keeps measurement costs down, while its elegance allows the flexibility to include a suitable level of expertise and sophistication to right size your measurement efforts to your marketing budgets.

Whichever measurement approach you choose, be sure to combine a sound methodology with the right expertise to learn what you need to know to make the right decisions. Measuring well will help you to sleep well and be a productive marketer!

Filed Under: Approach Tagged With: Effectiveness, Management, marketing, Measure, measurement, return on investment, ROI, scorecard

Wine Scoring & Marketing Measurement

February 16, 2012 By Rick Shea Leave a Comment

Tuesday evening I was browsing the latest edition of the LCBO’s ‘Vintages Release Catalogue’. This catalogue provides descriptions and sometimes wine critics’ quality scores for the new wine products about to be released through Vintages stores in Ontario. As I browsed, two thoughts came to mind.

Firstly, I noticed that most of the scores in this catalogue were between 88 and 92 on a 100-point scale. It struck me that this suggested the majority of the wines in this catalogue were of very similarly high quality, with almost all wines rated within a narrow 5-percentage point range. I found that odd, perhaps unrealistic, and decided to think about it. Secondly, I noticed that the wine descriptions were making me thirsty.

Wine Glass, Red Wine

Seeing the wisdom in choosing the beverage that best suited the task at hand, I poured myself a glass of red to compliment my thinking, sat down with the catalogue and made a few calculations and notes. Here are some highlights.

Vintages published scores for 57 of the 120 wines in this catalogue. The wine critics quoted used the 100-point scale for 48 of the 57 rated wines. The other nine were based on 20, 5 or 3-point scales.

Of the 48 using the 100-point scale, 41 (85.4%) received a score between 88 and 92, and 30 of those were either 90 or 91, which confirmed my first observation. The other 7 wines were rated higher than 92, leaving no scores below 88.

Taking a sip from my glass, I contemplated why so many wines received such similar scores, and how all this relates to marketing measurement. Here are a few thoughts:

Wine Scoring: The LCBO is in the business of selling wine and I suspect they have a policy of only publishing scores of 88 or higher. I tested this theory by looking at the two previous Release Catalogues and wasn’t able to find a wine scoring 87 or lower. Perhaps they’ve learned that lower scores reduce sales and so don’t publish scores below 88.

Marketing Measurement: Marketers are in the business of spending money effectively to drive positive business outcomes. Instead of measuring only the best marketing programs or those you might want to cast in a favourable light, measure and rank all programs so you can identify which are most and least effective, and then optimize future strategies accordingly.

Wine Scoring: By my rough count, the 48 scores using the 100-point scale were sourced from 26 different wine critics. While each used a 100-point scale, I have a hard time believing all 26 used the scale in exactly the same way. I also suspect that some critics are more generous with their scores than others, like my calculus teacher in CEGEP. The other important issue is that scoring wine is a highly subjective exercise. It isn’t at all uncommon for two or more tasters to disagree on a wine’s quality and the corresponding score. Experts have different opinions on subjective matters.

Marketing Measurement: To minimize inconsistencies, reduce or eliminate subjectivity and personal bias from your measurement processes. Having 26 experts using similar but sometimes different methods of scoring your marketing programs based on their personal opinions would not be a recipe for consistency. One person needs to lead your measurement efforts using one methodology that your organization understands and supports.

Wine Scoring: I did a little reading on wine scoring and discovered that wine critics can be inconsistent in the scores they award to the exact same wine on different occasions. For example, the influential wine critic Robert M. Parker has apparently pointed out that he sometimes assigns different scores to the same wine at different tastings, but that those scores tend to be no more than 3 points apart. It seems that differences in tasting conditions and the taster’s emotions can lead to different scores. To address this, I believe Robert Parker publishes average scores when multiple tastings produce different scores.

Marketing Measurement: Consistency is important in making comparisons meaningful. Pick one methodology that can be used consistently across all programs. Consistency should help you to avoid having all your scores cluster within a narrow range where differences may not be significant, or actionable. Programs can differ significantly in their effectiveness at meeting your objectives, and so their scores should reflect those differences. Also, if data for one metric is collected at various times, or from different sources, you might want to follow Robert Parker’s lead and use an average score for that metric.

Advice for Wine Drinkers: Don’t worry about the difference in quality between a wine that scores 88 and another that scores 92. Both are high quality wines and the difference in scores may come down to who tasted it, under what conditions and the tasters’ preferences. Here’s the fun part. Through trial and error, you should eventually be able to determine which wine critic your tastes best align with, and then the ratings and tasting notes from that critic will help you to make better wine purchasing decisions.

Advice for Marketers: Similarly, there will be some trial and error involved, but not nearly as much fun. Select a measurement methodology that you can apply fairly, without personal bias, and consistently across all programs. Be disciplined about measurement and it will ultimately highlight which marketing programs best meet your objectives and create value for your business. That will help you to make better marketing decisions, which you may wish to celebrate by opening a bottle of your favourite wine!

Filed Under: Approach, Objectivity Tagged With: bias, comparison, Effectiveness, Management, marketing, Measure, measurement

Do Daily Deal Coupons Work?

January 19, 2012 By Rick Shea Leave a Comment

Daily deal coupons are all the rage these days, led by Groupon and their numerous competitors. Since I have a pretty extensive background in consumer promotions, particularly couponing, I tend to get pulled into discussions on daily deal coupons and I often notice the frequent related media coverage. Setting aside all the buzz around them, they’re really just a new twist on an old marketing tactic, but do daily deal coupons work?

Traditional paper coupons and price discounts have worked quite nicely, since the late 1800s in the case of coupons, and I suppose for centuries or millennia in the case of price discounts. One of the main reasons to discount is to attract new customers to try your products or services, and then hopefully sell them more stuff and/or turn them into profitable repeat customers who will pay full price on future purchases. Daily deal coupons can certainly do this.

It is beyond the scope of this newsletter to address the dos and don’ts of daily deal coupons, but you’ll get some insights if you Google “Groupon horror stories” and “Groupon success stories”. I will say that you’ll want to have a way to limit response levels or to make sure you can live with whatever level of response you get.

To address whether daily deal coupons work from a measurement perspective, my answer leads to more questions, starting with “Well, it depends, what did you want them to do for your business?”

Drilling down a bit further, answers to the following four fundamental questions should inform the planning, objective setting and measuring of any marketing program. Clear answers should point the way to what and how to measure, and whether the program “worked” to meet its objectives.

Let’s look at each of these.

1. Who are you targeting? While many answers are possible, the best answer is often “new customers” to help grow your business. Merchants benefit most when they structure their daily deal coupon offers to attract new customers, rather than subsidize existing customers who would have bought without a coupon.

2. What do you want them to do? You’ll want new customers to buy for the first time whereas you may want existing customers to buy something more or different than usual. In addition, you might try to prop up an under-performing aspect of your business, such as your slow month or time of day, or a product that isn’t selling well.

3. How much value will that create for your business? This one is especially important and a bit tricky. You’ll need to consider the short term (this transaction) and the long term (the customer’s lifetime). Ideally, you’d like this transaction to create enough value to at least cover your costs, but if it doesn’t you’ll need to make up the difference and ideally much more over the lifetime of those customers who buy your daily deal.

Short Term – This Transaction: Consider your variable cost of providing the products or services you will sell through this coupon and compare that to the revenue from your share of the coupon selling price, which you will split (often 50/50) with the daily deal provider. Also, consider whether you’ll receive your share of the revenue when the coupon is bought, or when (and if!) it is redeemed.

Long Term – The Customer’s Lifetime: How long is the lifetime of a typical customer? On average, how many times will each customer buy over that lifetime, and how much will they buy each time? If each new customer’s lifetime is just this one transaction, it may not be worth your while to offer this coupon. But, if you can convert enough of those new customers to loyal repeat customers for many years, then discounting to get them in the door should be worthwhile.

4. How many people do you need to do that for this expense to be worthwhile? Once you know what a customer is worth to you over whatever time frame you want to use, and you know your costs, then you can set an objective for how much value (metrics include new customers, number of transactions, transaction values, etc.) you want this marketing investment to generate. Then you can measure against those objectives.

Clear Objectives Make Measurement Easier

To measure whether or not a daily deal coupon or any marketing program worked for your business, you need clearly defined objectives. In other words, to measure whether you have succeeded, you must first define success. Clear objectives will tell you what metrics to use and where to find the data.

It’s one thing to attract new customers with discounts, and quite another to keep them. Can you convert discount shoppers to loyal customers? Success will come down to your company’s ability to deliver a superior customer experience in the short term, and to build a positive relationship with each customer over the long term and maximize both the lifetime and the value of as many customers as possible. Of course, that’s something every business has to do well, however they find their customers.

Daily deal coupons may be a relatively new marketing tactic, but there’s nothing new about the fundamentals that determine whether you should use them, whether you’ll be successful and how to measure your success. Clear objectives will help you to decide whether to use daily deal coupons and to evaluate whether using them worked, however you define success. To compare the success of a daily deal coupon program to any other type of marketing program, well that’s a topic for another newsletter.

Filed Under: Approach Tagged With: Management, marketing, Measure, measurement, objectives, Planning

Emissions Test

December 15, 2011 By Rick Shea Leave a Comment

My car is getting old. Like me, it doesn’t always feel its age, but the reality is that by most measures it is getting old. One measure of its advancing age is that in order to renew my vehicle licence recently, my car first had to pass an emissions test.

In Ontario, when the Ministry of Transportation sends you your vehicle licence renewal application, they indicate if your car needs to be emissions tested. This time around, my middle aged car was due for a test.

The Ministry requires the test be performed at one of the accredited Drive Clean facilities listed on their website. I found a local facility and within 15 minutes of arriving, I drove out of there with the clean bill of health for my car that I would need to renew my vehicle licence.

While waiting for my car to be tested, I entertained myself by reading the wall poster and brochure that described the emissions testing procedure. As a guy who cares about good measurement practices, I was impressed by two things:

The Ministry has a consistent and transparent measurement process.
Their process includes a clear and simple scoring system.

Let’s look at how each of these illustrate what a good marketing measurement system needs to do.

Measurement Process

These are the steps that all Drive Clean facilities follow when testing a car:

Perform visual inspection
Drive car onto Dynamometer
Insert probe into tailpipe
Accelerate vehicle to 40 km/hr
Computer analyses emissions and compares to standards for make, model and year
Print test results
If Pass, owner can renew vehicle licence
If Fail, owner must repair car at a Drive Clean Repair facility, then re-test

Here’s what I like about this process, especially as the same things are true of a good marketing measurement process:

Its clear steps can be repeated consistently, making the results meaningful and comparable to other results, standards and benchmarks.

The process is well supported by documented procedures, such as the 20 page “Drive Clean Guide” that I found on line. The guide is amended and reissued periodically, as needed. Well documented and updated procedures ensure consistency in measurement and clarify who is responsible for doing what and when.

The standards that define passing and failing results are clearly documented.

Scoring System

The scoring system is quite simple. There are a few metrics for which your car must obtain a passing score. For example, on Hydrocarbons, as measured in Parts Per Million, my car needed to be at 66 or less in order to pass. Fortunately, my reading was 3 and thus my result on this metric was a “Pass”. (I lobbied for extra points for such a low score, but to no avail!)

The key elements of this simple scoring system that relate to marketing measurement are:

Metric: Define the metric (Example – Revenue) and the unit of measurement (Example – for Total Company, for Specific brand, per square foot, per average transaction, etc.).

Objective: Define the target result, or limit, or range of acceptable results. (Example – Average revenue per transaction of $50 for Brand X in 1st Quarter of 2012).

Result: The actual outcome for the metric and objective as defined.

Scoring: You need to score how good the result is as compared to the objective. Will you use a pass/fail, a 10 or 5 point scale, a score out of 100, or some other scale? Whichever you pick, be sure you can apply it consistently and fairly.

Summary

A properly documented measurement process will provide the consistency needed to make comparisons between programs meaningful. A clear scoring system gives you a way to convert all your results to a common metric, which also enables comparing programs to each other. These two essential components of a reliable marketing measurement system support making well informed strategic marketing decisions, which comes down to making choices between programs, refining your strategies and finding the most effective ways to market your products and services.

Filed Under: Approach Tagged With: Effectiveness, marketing, Measure, measurement, Metrics

Pictures at an Exhibition

November 17, 2011 By Rick Shea Leave a Comment

I’m a long time hobbyist photographer. While I occasionally get hired to take photos for commercial use (e.g. events, product shots, architecture), I mostly just do it for fun. In fact, I had never formally exhibited my photos prior to this past weekend when I participated in an art show on my street.

Now in its fourth year, this show is basically an art walk limited to the one block of the street on which I live. I was one of 10 artists exhibiting a variety of art in 5 houses, on both Saturday and Sunday afternoons.

Reflecting on the event Sunday night I decided that it was a lot of fun, I’m really glad I exhibited and I’ll probably do it again next year. Since that simple statement doesn’t make for much of a newsletter on marketing measurement, I thought I would elaborate on how I came to that conclusion.

What I Spent

To begin with, my cost to participate was very low. Since the walls in my house were already covered with plenty of framed photos and ready to exhibit, I just needed to shuffle a few around and decide on prices. I probably spent all of about $50 to get ready, including salty snacks and chocolate chip cookies for my visitors. So, my costs were minimal.

What I Sold

I sold a total of 7 photos for $525. I’m happy with this outcome because:

Over the 2 days, I estimate 45 people dropped by my house. One person bought 2 photos, meaning I converted 45 prospects to 6 customers, for a conversion rate of 13.3%. This seems like a good rate for an art show, but I don’t have a benchmark to compare it to.
My sales surpassed my expectations. I thought I would do well to sell two or three photos and maybe pocket $200.
My customer acquisition cost ($50 ÷ 6 = $8.33 per customer) was much lower than my average revenue per customer ($525 ÷ 6 = $87.50 per customer) making each transaction and the overall event profitable.

What I Learned

For some background, our marketing efforts consisted primarily of:

Post card invitations hand-delivered to homes within roughly a 2 block radius
Each artist emailed invitations to their own list of contacts
The organizers solicited and obtained support from local politicians who emailed local residents and tweeted our event
Free on-line publicity, most of which came through a new local website called GrownUps55plus, for which I’m saying thanks by this mention

Secretly wearing my marketing measurement hat while disguised as an artist, I randomly asked those visitors I didn’t know the usual “how did you hear about us” kind of questions. I learned that, in addition to those friends and neighbours I already knew, most people either:

Lived in the neighbourhood, or
Were friends of one of the other artists

From a product point of view, of the seven photos I sold, 4 featured trees as the main subject, 1 featured a tree and a window, 1 featured a window, and the other featured a racoon. Selling the racoon photo gave me a chuckle because earlier that day two other people had separately A) cursed the racoon, and B) saluted it with a middle finger. Clearly, one person’s art is another’s neighbourhood menace. Also, people like trees.

Another interesting fact is that the turnout was much higher last year when some houses estimated traffic at 300+ visitors. Perhaps that’s a sign people are less confident in the economy this year? Imagine if I could have applied my 13.3% conversion rate to 300 people instead of 45!

What I Would Do Differently Next Time

Based on my informal measurement efforts and observations, and input from the other artists, here are some thoughts for us to consider for next year:

Proximity and familiarity seem to bring people out, so we should expand our post card coverage beyond 2 blocks and each artist could invite more people.
Recruit more houses and artists to participate; the additional artists can invite their contacts, and a larger event with more art for sale should appeal to more people.
Continue to solicit free publicity and politician support.
Find a volunteer with public relations expertise, perhaps a student looking for experience, to help us get more local and on-line media coverage.

Conclusion

I didn’t apply measurement to our marketing efforts in any sort of disciplined way, yet maybe that’s appropriate for such a low budget event. Still, by simply looking at what I spent, what I sold, what I learned and what I would do differently next time, I should be able to improve my sales next year, and that’s what counts.

Whether for a small event like this, or a major marketing initiative, the best way to improve next time is to make sure you have a way of learning from what you did this time. Measure, review, reflect and capture what you’ve learned, so you can optimize future strategies.

There’s other soft data to consider, too. I had a great time, met some neighbours, saw some friends and enjoyed seeing people enjoying my photos. I look forward to next year and in the meantime, I’ll be out shooting windows and trees, or maybe even a racoon.

Filed Under: Approach, Effectiveness Tagged With: Effectiveness, Management, marketing, Measure, measurement

Separating Fact from Opinion

October 20, 2011 By Rick Shea Leave a Comment

Megan Kalmoe rowed for the United States at the Beijing Olympics and is a 2011 World Silver Medalist. While she’s obviously a great rower, I hadn’t heard of Megan before reading Randy Starkman’s article in the Sports section of the Toronto Star on October 13, 2011.

The story focused on the fact that Megan named two Canadian men to the 2011 edition of her “20 Hottest Male Rowers” list, which she recently published on her blog. Not surprisingly, I didn’t make the list. One reason is that I’ve never rowed in my life. There might be other reasons, perhaps many, but more on that later.

I’m happy when Canadians do well in any international ranking, but being a marketing measurement guy, my interest in this article was to learn about Megan’s rating and ranking process.

Here’s what I learned:

You must be a world class male rower
You may lobby Megan to get on her list
Megan’s female friends can nominate you and lobby on your behalf
If Megan thinks you’re hot, she might add you to the list

As far as I can tell, Megan’s approach is pretty subjective and unstructured, and the resulting list reflects her opinion, which is fine. I can’t tell whether lobbying influences Megan’s decisions, but it really doesn’t matter.

It is worth noting that Megan is just having fun with this and important strategic decisions aren’t being made because of her list. But, consider this. What would happen if the top three rowers on Megan’s list landed endorsement deals because they topped her list?

I think the fun and friendly lobbying might get a little more intense for the 2012 list, and Megan might feel the need for a more structured approach to minimize the impact of bias and personal opinion on the rankings.

For now, since this is all in good fun, Megan’s methodology for measuring male rower hotness is perfectly appropriate and as good an approach as any. However, when it comes to measuring marketing program hotness, marketers need a more rigorous approach.

What if you used a similarly unstructured method for measuring marketing programs? Brand managers would lobby you to have their programs highly rated. People working in sales, finance, customer service or operations would also offer their opinions. You wouldn’t have much fact-based data and you’d end up having to make an opinion-based judgment call.

The impact on your judgment call of everyone lobbying to influence your opinion might come down to:

The clout of each person doing the lobbying, perhaps related to their role in the organization
Each lobbyist’s communication skills and powers of persuasion
Your ability to separate fact from opinion, and to somehow remain objective

Here’s the problem. You don’t want the most effective lobbyists to skew the rankings in their favour. Nor do you want personal opinion and bias tainting your overall approach. Opinions are interesting, but not very actionable.

To be able to take action, make good decisions and adjust strategies, you need data to identify your most and least effective programs. A structured and disciplined methodology will give you that data, while filtering out opinion.

To remove as much personal bias and opinion as possible from your marketing measurement efforts:

Involve the Right People: Create a cross-functional group to pick an approach that balances everyone’s needs and interests, so the approach is fair and equitable for all.
Involve Unbiased People: People with no vested interest in which programs get the highest rankings could include an analyst, someone from Accounting, or an independent consultant.
Set Clear Evaluation Criteria: Disclose how you will consistently evaluate each program, so everyone knows and plays by the rules.
Set Objectives Up Front: This prevents people from later setting lower objectives than they would have up front, thus making both their successes and failures look better.

Whether you need to identify your 20 hottest marketing programs, or which types of marketing spending should be increased or decreased, make sure your measurement methodology gives you the unbiased and opinion-free data and facts you need to make better decisions.

As for the fact I didn’t make Megan’s list, I think if I was at least 20 years younger, 10 pounds lighter, 5 times as athletic, a lot hotter and a world class rower, I could have been a contender. Of course, Megan might have had a different opinion!

Filed Under: Approach, Effectiveness, Objectivity Tagged With: bias, Effectiveness, marketing, Measure, measurement, objective

Sabermetrics & Moneyball

August 18, 2011 By Rick Shea Leave a Comment

While browsing the Sports section of the Toronto Star over breakfast one morning last week, I discovered a baseball statistic I had never heard of before.

Cathal Kelly’s article about the Toronto Blue Jays focused on General Manager Alex Anthopoulus’ fine work in transforming the Jays into a ballclub with great young players and a bright future. In Kelly’s analysis, he also pointed out that the Blue Jays “already have the best player in baseball”, José Bautista, who “leads all of baseball in wins above replacement rankings.”

I’ve followed baseball since I was a kid, but this was the first time I’d heard of “Wins Above Replacement”. Since José Bautista is considered one of the best players in baseball, I guessed it was some sort of composite score that rates a player’s overall performance. Still, I wondered “What the hell is Wins Above Replacement?”

In doing my research, I read up on Sabermetrics, an approach to statistical analysis in baseball that emerged in the 1990s. I read about Billy Beane, the General Manager of the Oakland A’s, who is a leading proponent of using statistical analysis to aid in making decisions about which players to draft, trade or acquire. Billy’s work in this area led to a book being written about him called ‘Moneyball”. I also learned that many teams now employ statistical analysts and Sabermetrics experts.

Sure enough, Wins Above Replacement turns out to be a Sabermetrics statistic that estimates how many more wins a player would give his team as compared to a replacement player of below average quality. While there appear to be a number of formulae used by different Sabermetrics proponents, the principles they follow align nicely with my scorecard approach to measuring marketing programs.

Any Wins Above Replacement formula takes into account a number of batting and fielding performance metrics, weights those metrics appropriately, and tallies everything up to provide one overall score to assess a player’s contribution to team wins. Similarly, a marketing scorecard uses a combination of company, brand, customer and program performance metrics, weighted appropriately, resulting in one overall score to asses a program’s impact on the business.

They both:

Have a limited budget (for players or marketing programs).
Need to get the most for their money (spending efficiency and effectiveness).
Need to get better results than their competitors (win games, make money).
Need a way to organize and make sense out of lots of diverse data about the performance of their players/programs and team/company (so they can improve their player/marketing decisions).

This last point is the most important one. A general manager trying to assess the performance of different players on different teams, playing in different ball parks under different conditions, needs a way to organize and then convert all that data into one metric or score that enables rating and ranking players relative to their peers. Wins Above Replacement does that. It may be calculated in different ways by different teams, but so long as each team does its own calculations consistently between players and over time, their comparisons will be meaningful.

The same is true with measuring marketing programs. As a manager with a marketing budget, you need to evaluate the performance of different programs, with different objectives, for different brands competing under different conditions. A well designed marketing scorecard will give you a way to organize, evaluate and convert that diverse data into the single common metric you need to rate and rank each program relative to the others.

Pretend that you’re the Billy Beane of marketing and you need a decision support tool that helps you to organize and make sense of diverse data about a variety of marketing programs. With a scorecard based approach and your considerable wisdom, instincts and experience, you’ll gain the insights you need to improve your marketing effectiveness!

Filed Under: Approach, Effectiveness Tagged With: Effectiveness, Management, marketing, measurement, measuring, scorecard

From My Perspective

June 16, 2011 By Rick Shea Leave a Comment

I like walking. Many mornings, my walk takes me to nearby Monarch Park. Over the last few years, I’ve frequently taken my camera to the park as part of a photographic self-improvement exercise which involves photographing the same subjects over and over.

Going back to the same park repeatedly forces me to develop my ability to see and capture photos of the same subjects in new and creative ways. I’ve learned there isn’t one right way to capture any one subject, and there are usually many fine ways to capture the same subject.

Here’s what I’ve observed about the two main variables I have to work with; light and composition.

The light in the park can look very different in different seasons, at different times of day and in different weather.

As for composition, I see the best new photos when I change my perspective by changing where I’m walking or standing.

A recent foggy morning created new circumstances. The same old views looked very different due to the soft light and the masking effect of the fog. In search of a new composition, I changed my perspective by leaving my usual route and walking through a more wooded area. Through the combination of the fog and a different perspective, I quickly saw something I had never seen, which led to my capturing one of my favourite images of the park.

When I first developed my approach to measuring marketing, my perspective at the time was that I was trying to solve the following problem. I was trying to help marketers answer questions like “Did that marketing program work?” or “Did I use my money wisely on that program?”. From that perspective, I designed a scorecard to measure individual marketing programs.

I started using that approach but quickly realized that my perspective on marketers’ problem needed to shift slightly.

Instead of asking “Did that marketing program work?” marketers wanted to know “Which of my programs worked best?”.

Instead of asking “Did I use my money wisely on that program?” they wanted to know “What are the best ways for me to use my money?”.

The differences between the original and the revised question in each pair are small in words but large in meaning. Answering the original questions can provide some insights about individual programs, whereas answering the second questions goes well beyond those insights.

With my slight perspective shift came more clarity about the problem marketers need solved. I developed a more robust scorecard, using a methodology that could be applied consistently across all programs. That change enables marketers to compare programs to each other so they can see which programs are most and least effective, and then adjust their marketing strategies and improve business results.

Just as importantly, I created an effective process for identifying and ensuring the right things would be measured on that scorecard.

Light and composition are the two main variables that impact taking photos, while a marketing measurement system’s two main variables are the design of the scorecard and the choice of metrics to put on the scorecard. In both cases, there isn’t one right or perfect approach, and many will provide worthwhile results if you get the fundamentals right and focus on solving the right problem.

The Right Problem to Solve: The reason to measure your marketing is to optimize your marketing decisions and improve your business results.

Scorecard Design Fundamentals: It needs to be flexible enough to measure any kind of marketing program, while also consistently using a standardized methodology that makes it meaningful to compare each program to all the others.

Choice of Metrics Fundamentals: Understand how your company creates value, who your ideal customers are and how you define profitable customer behaviour. Your marketing should target those customers and that behaviour, and the metrics you chose should help you to see whether marketing is helping to create value for your business.

Measurement is an integral part of continuously improving your marketing effectiveness. With a steady effort, an occasional shift in perspective and an eye on the fundamentals, your measurement will evolve and improve over time, as will your marketing. In the meantime, I’m here if you need my help, unless I happen to be out in the park changing my perspective.

Filed Under: Approach, Metrics Tagged With: Effectiveness, marketing, measurement, Metrics, scorecard

Opportunity Knocks!

May 19, 2011 By Rick Shea Leave a Comment

Last Saturday at around 5pm I was frantically cleaning my house. I had cleaned the bathrooms, vacuumed, swept and dusted, and was about to wash the kitchen floor when it suddenly hit me. I was wasting my time.

Sensing an opportunity, I wisely settled into my favourite comfy chair, put my feet up and took a nap. This was a much better use of my time than washing the kitchen floor, especially considering the night ahead. Here’s why.

At roughly 8pm that night, the first of 50 or so of my friends would begin knocking on my front door to attend my annual spring party. I knew that many of the 50 would gather in the kitchen. All those feet would be guaranteed to make for a dirty floor, which I would have to wash again after the party.

The additional benefit of washing that floor before the party would be negligible, at best. I’d feel good about my clean floor (which no one else would notice), but only until all those feet arrived (with friends attached) and began to mess it up. On the other hand, a nap would really boost my energy for the evening.

Excessive investments of time and energy into house cleaning prior to a party are adversely affected by the law of diminishing marginal returns. (Try quoting me if you need to get out of a cleaning chore sometime!) For each extra cleaning investment, you get less and less back in terms of the quality of the party or the guests’ enjoyment of it. While a house needs to be clean enough to be presentable, it doesn’t need to pass the white glove test.

Similarly, investments of time, money and people into marketing measurement are also impacted by diminishing marginal returns. You shouldn’t overspend on measurement and it doesn’t need to be perfect or pass the measurement equivalent of a white glove test. It just needs to be good enough to help you to make better decisions. Consider the following visual:

The vertical axis represents the resources you invest (money, time, people) to measure your marketing. The horizontal axis represents what you learn from those measurement investments that help you to make better marketing decisions, thus improving your marketing effectiveness.

The curve represents my view of the rate at which incremental measurement investments improve marketing decision quality. Generally, the more you invest in measurement, the better your marketing decisions get, but it’s not a straight linear relationship.

Let’s look at this curve in each of the three zones separated by the two red horizontal lines, starting from the bottom zone.

Bottom Zone

Characteristics: Starting at zero on both axes, as you begin to measure you very quickly learn things that can improve marketing decisions. Most organizations in this zone have very small marketing budgets, and few resources, so it may not be possible to invest much in measurement, nor are there many marketing decisions to improve.

Recommended Strategy: Take advantage of no or low cost measurement tools and internal data. Measure anything and you will likely learn something useful.

Middle Zone

Characteristics: Organizations in this “Opportunity Zone” have marketing budgets that are big enough to be worth measuring, and can allocate a small percentage of their budget to measurement. The opportunity in this zone is that small investments pay off quite nicely in the way of improved marketing decisions. The return from better decisions shows up as lower or more efficient marketing expense, and higher revenue and profit.

Recommended Strategy: Consistently apply a disciplined and practical approach to learn what you need to know to improve decisions. Resist the temptation to over invest in the more sophisticated (and expensive) measurement solutions that will bring you closer to the steepening section of the curve where you get a lower return for your incremental investments.

Top Zone

Characteristics: Here we see the most severely diminished marginal returns from measurement investments. It takes significant additional investments to yield even the slightest improvements in decisions. Only the largest of organizations with enormous marketing budgets can play successfully in this zone, as small market share gains and sales lifts can be very profitable. Other characteristics of this zone will include a lot of complex data and sophisticated measurement techniques.

Recommended Strategy: Question every bit of measurement spending. Just as there is great opportunity to learn at the lower end of the curve, on the upper end there is equally great opportunity to reduce measurement costs without significantly damaging decision quality.

That’s the way I see the relationship between measuring your marketing and how it helps you to get better results. There are exceptions to every rule, but the law of diminishing marginal returns is one of economics’ most powerful laws, so ignore it at your peril.

Have you identified which zone of the curve you’re in? There are a lot of organizations in the bottom and middle zones, who may not currently measure their marketing, or who aren’t happy with their efforts to do so. If that sounds like your organization, a great opportunity knocks at your door!

In the meantime, if you need me, I’ll be in the kitchen mopping the floor!

Filed Under: Approach, Effectiveness Tagged With: Effectiveness, marketing, measurement, return on investment, ROI

Social Media and Social Eating

April 21, 2011 By Rick Shea Leave a Comment

Somewhere in the middle of a Dim Sum eating frenzy last Sunday at Rol San in Chinatown, my friend Elliot pointed out that five of our group of seven sitting around the table worked in marketing. Despite the fact that marketers can be creative and some in our group of seven are rather artistic, we’re nothing at all like Canada’s renowned Group of Seven painters. After our efforts last Sunday though, I’d say we are a group of seven skilled in the art of social eating.

Elliot’s comments came during a discussion about how a certain academic institution appeared to be measuring the success of a controversial event they had publicized through their website and social media. In response to criticism of this event, they pointed to their number of subscribers, as if that somehow indicated a level of support for their controversial point of view.

Of course, just being a subscriber doesn’t automatically imply agreement with every point of view expressed. In this case, the number of subscribers was irrelevant. It would be more relevant to know the ratio of subscribers for vs. against the event taking place, and/or the point of view being presented.

Around the table we began discussing how to measure social media and quickly agreed that volume or Activity metrics aren’t as relevant as metrics that track customer Engagement. Even more important to track is a third group called Conversion metrics. To illustrate, let’s look at these three types of metrics in the context of measuring social media and also our customer experience at Rol San.

Activity Metrics

Social Media: Examples include number of subscribers, followers, followers/following ratio, tweets, fans, and links clicked. You can get a sense of what people are doing, but less about why or how they’re feeling.

Dim Sum Customer Experience: Examples include the total plates ordered, the average items eaten per person, and the average revenue per person. These metrics would tell Rol San how much we ate, but they wouldn’t know whether we were satisfied customers.

It is generally more relevant to look at:

Engagement Metrics

Social Media: Examples include forwards, mentions, likes, comments, retweets and the sentiment of comments, tweets and blog posts. These types of metrics can provide more insights into what your customers are thinking and feeling about your brands and marketing programs.

Dim Sum Customer Experience: Rol San might want to know if that second order we placed repeated any items from our first order. (It was hard to tell amidst the flurry of plates and chopsticks.) Did anyone tweet or blog about our meal, or post a review somewhere? Were the comments or reviews positive or negative? Are the people who posted comments influential with the right audience?

It can be hard to tell what customers think and whether they are truly satisfied. That’s why so many eating establishments include a customer satisfaction survey with your bill. Many of these direct you to a website to give your feedback, which can then be linked to your transaction (what you ordered, your server’s name, etc.) to help round out the customer experience picture.

Still, engagement metrics and customer satisfaction scores have their limits. What customers say can often be different from what they actually do. Attitudes and opinions can help to predict behaviour, but all that investors, shareholders and bankers really care about is profitable customer behaviour, and how that behaviour converts into value for the business.

Conversion Metrics

Social Media: The greatest Conversion metrics of all are revenue and profit. Other examples include qualified leads generated, content downloads, registrations, reservations and orders; basically anything that might track key steps in acquiring, keeping and cultivating profitable customers.

Dim Sum Customer Experience: Rol San should care about whether we come back as a group, or individually with more friends, and whether we recommend to others to dine there. In a retail business, these metrics can be hard to track, which is one of the reasons loyalty and viral marketing programs exist, to both incent and track profitable customer behaviour. It’s also why hosts or greeters sometimes ask “Is this your first time here?” or “How did you hear about us?”

I can’t speak for the others in our Group of Seven Social Eaters (G7SE?), but I think I will probably return to Rol San someday. How’s that for mildly positive sentiment and uncertain repurchase intent? Rol San could invest a lot of money trying to predict my behaviour, but even I can’t predict what I’m going to do. They’d be better off tracking what I actually end up doing.

Conversion metrics are the most important metrics to track and they should be more heavily weighted on your scorecard. At the same time, don’t ignore Activity and Engagement metrics, as they are predictors of conversion. They can help you to identify where programs are succeeding and failing in creating the customer behaviour that leads to profits.

Why am I hungry?

Filed Under: Approach, Effectiveness, Metrics Tagged With: Effectiveness, marketing, measurement, Metrics, scorecard