In search of ROI

John Wanamaker, retail baron of yesteryear, famously found the ROI of advertising unmeasurable

I started to write about agencies and client procurement departments, but I got so hot under the collar I had to stand next to the air conditioner for five minutes to cool down. Then I realized the subject was too big to tackle in a single blogpost. So in this one, I’ll focus on what should be one of your pet procurement peeves: return on investment (ROI) questions, and how to answer them.

First, a little background for those who don’t pitch a lot: procurement departments, or ‘strategic sourcing’ as they often call themselves, now manage the agency selection process for most major companies. (By the way, who would ever want to be ‘sourced’: “Honey, I’m going to the grocery store to source something for dinner.” And ‘procurement’ is even worse—isn’t that what The Emperor’s Club V.I.P. was allegedly doing for Eliott Spitzer?) They’re staffed with purchasing agents, and often the same individuals who manage agency selection also contract for the raw materials, pens, toilet paper, etc. of their companies.

Once upon a time when I was new to this business, cronyism, the ‘old-boy’ network, the favor bank, and a decidedly ad hoc process dominated many, if not most, pitches. And in the days when marketing directors and product managers chose agencies, the selections were not always in the best interests of their company’s stockholders.


Not for nothing did one agency name its conference rooms after its favorite clients. And according to unverified scuttlebutt, another agency in the ‘80s and ‘90s was widely believed to provide keys to luxury automobiles and vacation hideaways to cooperative clients. Small price to pay for the chance to ride the tail of a blockbuster launch, the way some people think.

The pendulum eventually swung the other way, thank goodness. These transgressions understandably gave rise to today’s procurement process, in which supposedly objective, disinterested third parties—the aforesaid strategic sourcing people—mediate agency choice. Today’s stockholders are better protected, and the playing field is a lot more level for the agencies involved.

Trouble is, procurement doesn’t know the territory


Unfortunately, choosing agencies isn't the same as choosing toilet paper. That means, in salesman’s talk, procurement professionals often don’t “know the territory.” Yet the processes, technology, and the thinking many procurement people apply to the one is often the same as what they apply to the other. (If you’re thinking what I’m thinking, don’t go there.) The result has been a tremendous amount of frustration on the part of agency new business people, trying to figure out how to respond to all manner of irrelevant, misguided, and otherwise out-to-lunch questions in the requests for information (RFIs) or requests for proposals (RFPs) that open or close the gates to a pitch.

Even worse, a little knowledge is a very dangerous thing (or think, as I like to say). Take ROI, for example.

Somewhere, somehow, an idea took hold among procurement people that advertising, being a significant investment of company money, should be accountable and measurable. On the surface, it makes sense. An investment of X dollars in an advertising campaign should ultimately yield Y dollars in increased revenue. Otherwise, why spend the money?

Of course, advertising professionals from the time of John Wanamaker, the eponym of the now-defunct retail empire, have struggled with the practical elusiveness of this concept. Wanamaker's plaint, “Half the money I spend on advertising is wasted; the trouble is I don't know which half,” has echoed in one form or another in the halls of thousands and thousands of corporations. It’s also given rise to wave after wave of chicanery, fad-ism and outright fraud among charlatans proclaiming that they have the methodology or machine to answer the question. From eye-tracking to hypnosis to brain imaging, someone will try to sell a gullible client gizmatology claiming to break the code of what does and doesn’t work in advertising.

Fact is, in 2008, the state of the art is that it's very difficult to measure ROI on advertising, and agencies aren't the ones who can or should do it. One major pharma company has a 20-person Management Sciences team of PhDs and MBAs devoted to crunching secondary and primary research data (most of which agencies don't have access to), attempting to gauge ROI on their ad campaigns, sales efforts, Med Ed programs, and similiar. They come up with numbers, but rarely without a lot of controversy over their accuracy and projectability. theThe Advertising Research Foundation (ARF), which routinely and repeatedly studies this question, essentially shrugs its shoulders and says that the only strong correlation consistently established between mass media advertising and sales come from measurements of how much audiences like the ads. Just about everything else is statistical noise. Media selection is a bit better, because there are ways to estimate eyeballs on pages and viewership of programs, but even that is art as well as science. We know that advertising can produce results, but there’s so much else going on—PR, salespeople, word-of-mouth, etc—that technology rarely can tease out just how much of sales revenue results from any ad campaign, sales piece, or event.

That doesn’t stop procurement from asking for ROI. All the time. In RFP after RFP I’ve seen or responded to, agencies are asked, “How do you determine the ROI of advertising you do to health care professionals? To consumers? Of broadcast advertising? Of direct selling materials?” Etc.

Then they ask the agencies for case studies with ROI results.

Fact is, agencies (except for direct response agencies) not only can't do good ROI measurement; they shouldn't do it.
And procurement people shouldn't put them in the awkward situation of being asked for it.

Why can't agencies do it? Because it takes manpower, budgets, and skills outside the scope of how agencies staff, get compensated, and maintain as core competencies. Until the same procurement people put extra tens of thousands of $$$s into budgets for the extra task of measurement, it's fiscal folly to do so.


Agencies also can't do it because they only get a thin slice of the data. They're not routinely plugged into the client's sales numbers and syndicated surveys (although a case could be made to do so.) Most often, data are shared only on a need-to-know basis. So they're usually working in a marketing-data vacuum, with too little to work from to draw meaningful conclusions.

Why shouldn't agencies do it? Because they're interested parties, with a stake in the outcome. No agency wants to show its work isn't pulling its weight. So there's unavoidable bias in assigning them the task of self-measurement. Give that task to a disinterested third party, preferably a competent marketing research firm with a hefty budget to work with.

And the agencies shouldn't answer questions about ROI of client campaigns because that's unethical. Those results, if available, represent the intellectual property of the client. It's not the agency's to share.

Solving the conundrum

It’s truly a conundrum when you’re asked these questions, because you know someone is going to read the answer, knowledgeable or not. Do you ‘fake it’ by claiming your campaign, program, or idea was responsible for any sales increase over the time it ran? Some agencies choose this as the safest way, because it throws a sop to the naif who asked the question, even though others find it outrageous to attribute sales increases to your campaign or program idea alone. I suspect clients’ files of RFP responses are afloat in this kind of hogwash, though, and I’ve certainly contributed my share over the years.

Slightly more helpful, I think, is to gently attempt to wean RFP creators away from their erroneous ways. While you can’t fit an explanation of research methods, observational bias, and agency compensation into a 500-character box on an Ariba© online RFP form, you can briefly say that ROI evaluation methods are both confidential to individual clients and best handled by objective third parties—and that you’d be delighted to partner with the client in developing best practices.

I don’t usualy advise clients to leave the answer blank or put “Not applicable”. That suggests that you overlooked the item, or don’t care about the clients’ interests in this area.

Dialogue for lasting results
Like the best medicine, prevention is the best solution to ROI problems in RFIs and RFPs. If you discover your clients’ procurement people are asking lots of these questions, try to set up some time with their senior-most people to talk about the issue and share your perspective, experience, and knowledge. Talk about agency accountability—how should it be measured, if ROI is unmeasurable? Who should do the measurement work? How much of the role is the agency’s, and how should it be funded (overhead, part of fee, etc.)? Send them a copy of this article, with a note saying something like, “Interesting point of view—I’d love to talk about this with you.” (You might want to edit out the ‘air conditioner’ introduction first.)

It could lead to a broader conversation, about how agencies can help front-line procurement people better understand the services for which they’re contracting, and how to make agency-client relationships more partnerships than adversarial relationships. That would be a great conversation to have.

---