Fish just wanna be fed: "$6 Billion in Online Holiday Sales by the End of This Month!$24 Billion in Internet Ads by 2003! $2.3 Trillion E-Biz Predictions by 2010! The virtual science of high-tech forecasting. By JIM FREDERICK
You don't need an expert to tell you we're living in the middle of a technological revolution. Everybody knows
that the Internet is a
big deal and getting
bigger, and all those
claims that its
significance rivals that of the telephone, the railroad, the television and the
automobile don't seem as delusional as they did a few years ago. Still, it's
easy to wonder, to want to know, to need to know, exactly how
momentous the Internet's impact will be. How big will, say, the online
advertising industry get over the next three years? Just how many people
will trade stocks online? And how much money will be spent shopping on
the Web this holiday season, anyway?
The most honest answer to all of these questions is, of course, There's no
way to know. But that's pretty unsatisfying, isn't it? Just as nature abhors
a vacuum, so, too, does the human mind despise ambiguity. So let's try
these answers instead: online advertising will be a $24 billion industry by
the year 2003, 20 million households will trade stocks online that same
year and consumers will have spent $6 billion online from November
through the end of December.
Says who? The experts. The experts, that is, at Jupiter Communications
in New York and Forrester Research in Cambridge, Mass. -- the two
most prominent companies to have recognized America's
Internet-certainty shortage. Stepping boldly into the answer vacuum, the
companies have transformed themselves into prediction factories,
supplying the world with a steady stream of airy, context-free, yet
reassuringly precise projections like the ones above. Of course,
seemingly exact yet largely unconfirmable statistics pop up all the time;
we seem to need them somehow, so we simply accept it when told that
70 percent of all filed pieces of paper are never looked at again or that
the average man thinks about sex four times a minute. But the headlong
rush of the Internet, and the fortunes to be made or lost because of it, has
created a particularly sharp craving for a numerical accounting of what
will happen next. So who can blame Forrester and Jupiter for recognizing
that uncertainty is just another demand in search of a product?
As for the companies themselves, they certainly do nothing to contradict
the idea that they are virtual oracles. As one Jupiter analyst puts it, "We
define a worldview, and everything we say has to fit into that worldview
-- kind of like the Catholic Church."
Flip open any major newspaper or
magazine on literally any day, and you will
probably encounter a headline or a statistic
on the order of, "Online grocery shopping
will be a $783 million business by the end
of 1999." And Jupiter or Forrester will
almost certainly be cited as the source. Where do the numbers come
from? "You can build a forecast in about 10 seconds by sitting in a room
and guessing," says Evan Cohen, the director of Jupiter's data research,
between bites of a footlong chicken-and-cheese hero. "Sometimes I fear
that folks think that's how we do it. But we have extensive models built
for these kinds of things."
This is true, although those models have little resemblance to the
statistical techniques, like regression analysis or time-series analysis, used
by traditional market researchers or industrial forecasters to determine
next year's worldwide consumption of, say, gasoline or Coca-Cola. That,
of course, is because enterprises that forecast the consumption of
resources or consumer items can draw on decades of historical data.
Using such techniques, Cohen notes, "would be nonsense for us, given
there's no real historical patterns -- we're dealing with an environment
here that has existed for two or three years." (And even sophisticated
techniques relying on extensive historical data are hardly infallible.)
To understand how Jupiter and Forrester
come up with their numbers in the absence of
the kind of raw data available to analysts in
more mature industries, consider Jupiter's
assertion in November that American
consumers will spend $78 billion shopping
online in 2003. As most projections at Jupiter
do, this one began with telephone surveys
conducted in partnership with a traditional
market-research company called NFO
Interactive. Data was collected on who is
online now and on how much they probably
spend, and the extrapolations common to all
polling data were made. Those who aren't
online were asked when they might be; people
were quizzed as to how much they plan to
spend online in the years ahead, and on what.
Jupiter then asked more than 200 retailers, in
formal interviews, about their sales
expectations.
The problem is that retailers tend to be overly
optimistic about their forecasts and that most
consumers can't really know how much they
will spend on, say, airline tickets in 2001. So
Jupiter compared the estimates it got for online
shopping with the historical growth rates of
similar ways to shop (like catalogs and QVC)
and the total size of various retail markets by
category. Using these elements to "triangulate
toward market sizings," as a recent research
report puts it, the analysts estimated that the
number of Americans who buy stuff online will
grow from 18.8 million in 1998 to 85 million in
2003.
At Jupiter, the world of things you might buy
over the Web is cut into 28 product categories
-- movie tickets, furniture, groceries, etc. --
whose suitability for sales online is ranked in
13 criteria (like "ease of delivery," "product
variety" and "ease of evaluating online"). The
rankings are pretty much subjective; the
analysts assign each category a relative score
that they figure makes sense. When that's
done, Jupiter tries to marry the number of
people shopping in each of these 28
categories with how much they will spend.
Again, the firm concocted a qualitatively
scored grid to evaluate how much of a
person's total annual expenses in a particular
category will shift to the Web over what
period of time. Take that dollar amount and multiply it by the number of
people you think will be shopping in that category, and you have an
estimate of how much will be spent in that category every year through
2003.
At that point, Cohen says, "we go through a process of gut checking."
That is, they look for the categories with conclusions that are obvious
nonsense. "Once you develop that final number, you can then compare it,
for example, to the overall size of that market in the U.S.," he continues.
"At first glance, you know, you'll look, and all of the line-by-line
assumptions will make sense, but gosh -- the total implies that 50 percent
of the hotel market will be online? Well, that doesn't make sense." And
when that happens, Cohen explains, they throw out the results, go back
into the score cards and retool them, sometimes repeatedly, until the
numbers do "make sense."
Only when that's done for each category can you add them all up and get
$78 billion spent shopping online in 2003. So when Jupiter employees
claim that many of their predictions take weeks to build, it's easy to
believe them. Nonetheless, the precise figures that these Rube Goldberg
forecast contraptions yield give little hint of the layers of extrapolation and
subjective guesswork underneath.
Take another prediction, this one from Forrester: global spending on
Internet ads will hit $24 billion in 2003. Charlene Li, the analyst who
headed the team that made that projection, notes that advertising is a
fairly straightforward business-to-business transaction and that data from
years past has shown that fewer than 100 sites capture more than 90
percent of Internet advertising. So rather than build minutely detailed
computer models of past and expected behavior, the bulk of her team's
research involved phone interviews with 100 or so executives in charge
of buying and selling ads, asking how much they expect to spend on Web
advertising over the next four years and how much ad revenue they
expect to receive from their own Web properties. Those numbers are
poured into a spreadsheet, fiddled with to account for exaggerations and
overlaps, Li says, and then "we just converge on a number."
No one at Forrester or Jupiter claims that what he or she does is science.
Instead, the companies simply maintain that while their methodologies
may not be perfect, they are more thorough than anyone else's.
"Interviewing 50 people is not statistically valid," concedes Frank Gillett,
a Forrester analyst, discussing a favored Forrester research technique.
But "somebody has to pull more than 50 people out of their hat to say
that they know better."
ive years ago, Jupiter was a modest trend-watching newsletter
operation with a dozen employees, but as the Internet has
exploded, so, too, has Jupiter's business. Jupiter now employs 250
people, occupies two floors in a cavernous loft building at Broadway and
Houston and recently went public in an offering that raised $66 million.
"We're the first and last word on what people should be doing and how
businesses should be comporting themselves," says Gene DeRose,
Jupiter's C.E.O., in his modest corner cubicle sprinkled with rebel
artifacts like a guitar, a stuffed Jerry Garcia doll and a poster from the
Knitting Factory. "Our goal is to get thousands of companies to trust us in
their decision-making process."
Over at Forrester's headquarters in Cambridge, George Colony sits in
one of the office's many small meeting rooms named after rock stars like
the Allman Brothers (his favorite band), leans back in his chair and stares
at the sudden downpour outside. Colony founded Forrester in 1986 to
study the effects of the popularity of the personal computer. Despite its
original orientation toward more nuts-and-bolts technology topics,
Forrester has successfully retooled itself to address the Internet almost
exclusively. "We do not sell information," Colony says. "We sell objective
advice. We sell ideas. We are an idea factory. We are futurists. We are
always looking toward what's next."
More specifically, Forrester and Jupiter are
so-called syndicated research firms. This
means that they publish a wide range of
reports with extremely high subscription fees
to a small, targeted audience of corporate
executives. For about $20,000 you can buy a
subscription for one of eight color-coded
subjects, or "practices," from Jupiter
Communications. Then, every month for a
year, you'll receive a 16- to 32-page report
filled with analysis and advice, survey data and
industry forecasts. A subscription also includes
one-page, news-oriented weekly updates and
phone time with analysts. Forrester's pricing
plans are a bit more complicated and variable,
but the gist is the same. As sidelines, the
companies also provide a little one-on-one
consulting and hold periodic high-ticket
industry conferences at large hotels in places
like Miami, Boston, San Francisco or New York.
What happens when they're wrong? "We are like Eisenhower's
weathermen," Colony says. "When we are wrong, young men die." Well,
not really. In fact, Forrester and Jupiter have made their shares of
high-profile blunders. Forrester, for example, claimed a few years ago
that AOL's membership would top out at 6 million and then recede; it's
now 19 million. Meanwhile, remember Jupiter's prediction from 1997
that online grocery shopping would be a $783 million business by 1999?
The real number, $230 million, was off by more than a factor of three.
Jupiter came up with its number by way of the same process that it used
to predict, more recently, that $78 billion total shopping figure by 2003.
What went wrong? "The whole middle part of the grocery-delivery chain
is a huge issue," DeRose says. Jupiter knew that at the time, but believed
that somebody would "come up with an infrastructure" to solve the
problem nationwide. That still hasn't happened. Last year, Forrester
underestimated the growth of the number of people coming online
because the company didn't foresee the emergence of the sub-$700 PC
that included Internet access.
Now ask Shelley Morrisette, who until recently served as Forrester's
director of quantitative research, about the consequences of such gaffes.
Accountability? He guffaws at the notion, practically shouting: "There's
none! There's none!" For 22 years, Morrisette did quantitative research
for Gallup. "We prided ourselves on being exactly right," he says. "When
you made a mistake, if you called an election and picked the wrong
candidate or something like that, you were beat over the head."
In the Internet-prediction business, no one ever seems to get beat over
the head, probably at least in part because, so far anyway, winning or
losing has proved to be a much murkier concept in e-commerce than it is
in an election. Morrisette says he wanted to address more upfront and
publicly the uncertainties involved in making such predictions -- for
instance, by releasing a "band," or range, of numbers, instead of one
precise figure -- but was often thwarted. "I said by the year 2003, the
number of households doing online financial services would be between
17 and 22 million households, or whatever it was, depending on what
would happen, and I gave a list of the things that might happen," he
recalls. "I got slaughtered. They said: 'We need one number. If you don't
give one number, it looks like you're equivocating.'"
Among those who far prefer something definitive, told to them by an
expert, are journalists. Consider this example: not long ago, a reporter for
a well-known national magazine was writing an article about all the
problems online brokers were having just keeping their Web sites up and
running. After researching the problem with various sources, and after
almost having finished writing the article, this journalist still needed some
numbers to buttress some of his claims, and maybe a quote or two to
back up his conclusions. So he put in calls to Forrester and Jupiter.
This reporter was, of course, me. And at the time, I didn't give it a
second thought. I was on deadline and I knew the drill: someone at one
of these companies would get me what I needed fast. A Forrester
analyst, Michael Gazala, it turns out, called me back first. From our brief
conversation, I got a quick statistic of how many households were
expected to be investing online by the end of 1999 and a handful of
snappy quotes. In fact, Gazala seemed willing to riff on the same thought
a number of times, until he sensed that he had delivered the perfect
quote. He also seemed to know that mainstream journalism often
precludes the writer from making an observation in his own voice. So
rather than simply saying "Yes" or "You're right" after I volunteered an
opinion, he would, without prompting, repeat my conclusions back to me
in complete, eloquent, usable sentences.
Now, I'm not the only journalist who has ever done this; practically all
journalists need experts in their stories, which is why Forrester and
Jupiter pop up in just about every media outlet you can think of --
including this magazine. The point is, Forrester and Jupiter know the
press needs them. Both companies aggressively promote their analysts as
go-to quote machines through a steady stream of news releases, press
hot lines ("Thank you for calling Jupiter Communications, the worldwide
authority on Internet commerce," goes the main switchboard's greeting)
and job profiles dictating that phone time with reporters is an essential
part of every analyst's job. Jupiter, which has 50 research analysts,
receives 300 press calls a week. Forrester analysts were quoted more in
the first six months of this year than in all of 1998. That impressive tally
will translate directly into dollars for Forrester analysts, whose individual
bonuses are pegged, in part, to whether or not they meet quotas of major
media mentions.
What's more, many of these analysts are perfectly willing to blithely offer
a confident take whether they're experts on the subject at hand or not.
Charlene Li, who covers Internet commerce for Forrester, says she fields
anywhere from 5 to 20 calls a day, on all kinds of topics: "Like deals. I
get a lot of calls when there are big mergers. If Lycos buys Quote.com or
if there's a new free I.S.P. service, I'll get calls. A lot of times a reporter
will call about this deal and say, 'Is this a good deal?' and I will go,
'Hmmmm,' and look it up, or ask them to tell me a little bit about it."
So what if they ask for an opinion on something she knows nothing
about?
"I give an opinion," she says. "I was scared at first. It took me a little
while to get used to that. But you give 'em an opinion."
From the companies' point of view, there's a good reason to get quoted
saying something definitive. Josh Bernoff, Forrester's man on the
interactive-TV beat, declared earlier this year that personal video
recorders (gadgets that are essentially more flexible, easier-to-use
VCR's) would mark "the end of network television." In person, Bernoff is
a bearded, heavy-set, quiet-spoken guy who looks quite at home in the
academic environs of Cambridge. He is also thoughtful, whip smart and
well aware of every move he makes with the press. That now widely
quoted crack about the death of network TV? It turns out he didn't really
mean it. "I mean, there are still railroads, right?" he says now. What he
really thinks is that network TV will lose about half of its market within
10 years. That's an industry-transforming development, to be sure, but
it's not quite the end of CBS, NBC and ABC.
But, Bernoff explains, he's not really talking to reporters for the benefit of
the average reader. "It's a very deliberate thing," he says. "I want to say
something that would make a potential client say, 'Wait a minute, we
gotta talk to this guy.' If someone reads that, and it leads to a contract,
then I've done a good job."
uthority has been considered a currency built silently over decades;
now it is conferred overnight upon those with the largest
megaphones. No doubt Forrester and Jupiter have achieved levels of
expertise and enjoy good reputations because they were among the first
to recognize Internet commerce as a significant business development.
And these days, executives at both companies point to their own
business successes, and high client-retention rates, as proof of their
authority.
Many Internet executives, those who are clients of Forrester and Jupiter
as well as those who are not, tend to offer the companies lukewarm
praise on the record and lukewarm criticism off it. Publicly, they say that
they value certain kinds of reports or individual analysts. Privately, they
admit that they view the companies and the media froth they generate as
necessary evils. Tom Watson, editor of an online news service covering
Silicon Alley called AtNewYork, likens them to players in what he calls
the Kabuki dance of Internet business plans. "I have not seen a business
plan, and I see hundreds every year, without a Jupiter or Forrester factlet
in it," he says. "And every business plan out there is hogwash. Nobody
knows what their business is going to look like in five years, and
everyone who writes one knows that. And the people who are investing
in these companies know that. But you gotta have numbers in the
business plan, you just gotta have them, and Forrester and Jupiter
provide them, so you put them in."
Journalists rely on Jupiter and Forrester for quotes and figures for much
the same reason: we need answers. When I took Forrester's word for it
earlier this year that there would be 3.5 million households trading stocks
online by the end of 1999, it wouldn't have meant a thing to me if the
company named a figure double or even 10 times that amount. I just had
a blank to fill with an expert guess about the future, and I got it.
As for Jupiter and Forrester, they have -- to put it mildly -- a conflicted
relationship with the projections they produce. There are about four
stock defenses they offer for their numbers, each a different step along an
ideological retreat from the certainty implied in the numbers themselves.
In fact, you often hear a jumble of more or less strident defenses in a
single conversation. Sometimes, for example, executives vigorously
defend the accuracy and methodology of their predictions and hand you
a propaganda sheet touting their greatest soothsaying hits. Other times
they insist that clients are not interested in the details, so the difference
between $15 billion and $10 billion when you're talking years from now
-- it's all academic, isn't it? Still other times they will say that the numbers
are really only marketing and press-relations tools because big, specific
numbers are what journalists love. Finally, when asked outright if the
whole enterprise isn't somehow a little ridiculous, they will sometimes say
that the numbers themselves are irrelevant, that the research is not about
numbers at all but about acting as a catalyst for thought by the clients.
George Colony and Gene DeRose steadfastly maintain that the accuracy
of their companies' predictions is the bedrock of credibility. "As I like to
say, if we can bat .700, if we can get it right 7 out of 10 times, then we've
helped our clients," Colony says. DeRose echoes that sentiment, almost
eerily. "If you can maintain a healthy 60-to-80 range of being on the
money, you're doing the right job," he says. Fair enough. So, precisely
what percentage of the time are these companies correct, and how do
they keep track?
Well, they don't. Turns out, this is a convenient exception to the rules
about exactitude; their track records are, the two C.E.O.'s admit, nothing
but ballpark estimates. An exact measure of past accuracy would be
pretty much impossible, DeRose maintains. "It's a struggle to define what
the nature of accuracy is," he adds. Over at Forrester, Colony basically
agrees. "We used to track it," he says, as he turns to reassess the
downpour outside, "but I don't have any data for you. My feeling is, if we
didn't do any better than .700, we wouldn't have a business." |
