Contrasting Bandwidth Studies - Part 2 of 2
------
Internet backbone traffic to have been around 1,500 TB/month at year-end 1996, but this estimate is less solid.)
(E) On slide #13, it is estimated that prices are dropping in half
each year, while traffic is growing 4x, which implies revenues should be growing 2x per year. Yet we do not see such rapid growth in revenues. Generally, carriers' IP revenues from business customers seem to be growing at 30-50% per year. (See, for example, the May 11, 2001 report entitled "IP!," from J.P. Morgan and McKinsey.) Furthermore, since prices are not in general dropping in half annually (with the exception of some major routes where there is a lot of fiber and heavy competition), this claim seems very questionable.
(F) The claims about utilization of backbone links are questionable. Utilizations are low, and are likely to stay low for a variety of reasons. As an example, one can examine the AboveNet network, which makes its detailed traffic statistics publicly available at <http://west-boot.mfnx.net/traffic/>.
The above points contribute to my doubts about the soundness of Larry Roberts' conclusions.
4. Final concluding remarks
While I am skeptical of Larry Roberts' 4x growth rate estimate, it could conceivably be correct. If it is, then something very interesting must be going on. Since we do know that the bandwidth of local connections to the Internet leased by the ILECs is not growing very fast, this would most likely mean that the fiber provided by CLECs is finally beginning to carry significant traffic, and new applications are being deployed much faster than before.
What makes the 4x growth rate feasible is that the Internet is not
all that large, certainly not when compared to the amount of fiber in the ground. For example, with very few exceptions all ISPs have typically just a single OC48 or at most OC192 link along their major routes. Yet the facilities based carriers typically have between 40 and 800 fibers along each route, and each fiber is usually capable (with current DWDM technology) of carrying around 80 OC48 or OC192 wavelengths. Thus only a small fraction of the fiber capacity is currently used for Internet traffic. (Another way to look at this is to note that the estimate on slide #6 of Larry Roberts' PowerPoint deck of just 1,200 Gb/s sum of edge ports for the 19 ISPs corresponds to just about 120 OC192 router interfaces.)
Although I am skeptical of the 4x growth rate that Larry Roberts estimates, I agree with him on several important points. One is that it is business traffic that is dominant on the Internet. (Too many projections for the future of the Internet seem to be based on the behavior of residential customers only.) Another is that there is no sign of a serious slowdown in the rate of Internet traffic growth. There have been recent estimates from financial analysts that Internet traffic growth is on a rapidly decelerating path, and even one claim, by the head of Nortel, that not just the rate of growth, but Internet traffic itself, has gone down. In that view, the telecom crash was caused by users abandoning their former ways, and tempering their appetite for bandwidth. Neither Larry Roberts nor I see any evidence of that. I would not go as far as he does in calling Internet traffic "anti-recessionary," but there are factors (such as plummeting prices of transmission and switching capacity) that could stimulate traffic growth even in a recession. The tragedy of Sept. 11, 2000 can only spur the search for redundant paths, duplication of databases, etc., which will fuel growth. Internet traffic is growing rapidly (even 2x per year is rapid by any standard measure), and the crash was caused by the collision of reality with unrealistic expectations. Business plans made on the basis of assumptions of 8x or 16x growth per year, or on the basis of the associated myth of "Internet time," could not survive in an environment of 2x growth per year.
THE EARLIER EXCHANGE BETWEEN ODLYZKO AND ROBERTS
On September 15, 2001 we wrote to Larry Roberts: 'The Caspian Networks early August bandwidth study made no sense to me Larry. That is why I asked Andrew if he would critique it. You are asking to be believed as an article of faith in a study with no independently verifiable data and findings.....data and findings more over on which the market success of your product may depend. If you aren't saying, "trust me," it seems that your PR people are saying that. But as you know one problem in the establishment of trust is that trust simply isn't transitive."
On September 19, 2001 Roberts replied: Gordon, I do not know if you have heard me talk to the study or not. If not, that might explain whatever is making no sense. The method of computing the traffic from the raw data is quite simple and straightforward. If this is confusing please talk to me. There was for this initial version of the study 19 backbone networks and 3 data points per ISP. This creates quite a statistically sound data base where any errors in adata point from one ISP make little difference to the total. The overall growth rate is therefore quite well defined. It may not continue in the future at this rate, but the increase for 4/1/00 to 4/1/01 is quite clear.
I did include all the data points in the presentation so that no one
would need to just trust me. However, the names on those ISP's are very confidential to the ISP's providing the data and they fear that the stock market might mistreat them if their names were revealed. In some cases the utilization is not up to industry standard (40%) so that they fear that people may say they are wasteful, rather than realizing that they are building a OC-192 network because it is the only way to compete and are quickly filling it. Thus, I cannot reveal the names or I will lose credibility and future data with the ISP's.
As to Caspian's goals, we are supporting the process because we also need the data, but I doubt it will affect our sales. The ISP's either need better switches or they do not. The study does not change their decision. However, many of the ISP's are thankful that real data finally exists so that they can see where they are in the IP race. I have never said "trust me", I have been very clear, this is the data I have collected, it is all on the table, and it covers only the period measured. I would hope everyone uses it as they see best. Since there is so many incorrect projections from little or no data, I believe it is important to make real measurements and publish the data.
As to Andrew's comments [Editor: delivered to us on September 10, 2001]:
Odlyzko: This estimate goes counter to the estimates that Kerry
Coffman and I have made. We first observed back in 1997/8 that Internet traffic was growing only about 2x per year (by which weexplicitly said we meant growth rates of between 1.7x and 2.5x per year, since our data did not allow greater precision). This was extremely controversial, as it went counter to the almost-universally accepted "Internet traffic doubling every three or four months," which corresponds to rates of between 8x and 16x per year.
Roberts: The only published estimates of fast growth were those by UUNET saying that they were growing at 8-10x per year for many years. That was true, but was only them catching up with the total market, not the market growth. Everyone I know realized this and paid no attention.
Odlyzko: We further predicted that this was likely to be the natural rate of growth in the future, and in a study last year (updated this year) confirmed that prediction. (All these studies are available on my home page, at dtc.umn.edu
Roberts: This study correctly shows that the rate could not have been 8x since 1996 but with no actual IP traffic data, does not indicate anything about the 2000 or future growth rates. As Andrew and I have discussed, he had no access to real data outside of AT&T and I would point out that one ISP out of 19 does not let one predict anything.
Odlyzko: Although the Roberts study also comes in with growth rates far lower than the 8x or 16x per year that had been widely assumed, it makes a huge difference whether the true growth rate is 2x (as Kerry Coffman and I estimate) or 4x per year (as Larry Roberts claims). If 4x is the right growth path, then the Internet industry will soon see a huge increase in revenues, and this will feed back into huge increases of sales in equipment. However, if the growth rate is 2x per year, then the picture is much less cheerful, as technological progress, combined with competition and the overbuilding during the past couple of years, would keep revenue growth to modest levels for quite a while. In particular, which of our estimates is correct is likely to become very evident within a year or two, from watching what happens to the carriers' revenues.
Roberts: Note that the price per terabyte that ISP's collect has
been dropping by about 50% per year. With prices dropping at 2:1 and traffic growing at 2:1 there would be no revenue growth. This has not been the case and would lead to an impossible situation in the future. The reason for this price decrease is DWDM, and strong competition. However, still switch vendors would still see a 2:1 increase in sales each year which would be fine for them.
Odlyzko: I am very skeptical of Larry Roberts' estimate of 4x annual growth. All the data I have seen is far more consistent with 2x growth rates. (Moreover, the recent cutbacks in capital spending plans by many major carriers for the next couple of years suggest that they do not see huge growth rates in traffic on their networks.)
Roberts: What is this data? I have read all his work as well as all
the published projections (many based on his work) and none of them ever looked at any IP traffic on any carriers network except perhaps the unstated use of AT&T data by Andrew. All other projections are based on secondary information like Nortel and Cisco sales, hardly a measure of the underlying traffic.
Odlyzko: I will not go into the details, but my papers with Kerry
Coffman are fully documented, and based primarily on publicly
available information, or else information where the sources are
explicitly named. Furthermore, all the evidence that has been
accumulating since our work goes to support our estimate of annual 2x growth. (For example, the plot of Genuity's traffic, recently made available at
<http://www.ima.umn.edu/talks/workshops/8-6-7.2001/serr/serr.htm>,
shows a very steady 2.2x annual growth rate for that carrier for the last 3.5 years.) One may ask how this 2x estimate fits with the growth rate of AT&T's Internet traffic, which was recently revealed by an official spokesperson to be around 3x per year (see the August issue of "The COOK Report on the Internet"). Well, Kerry and I knew the AT&T growth rate (although we could not publish it), but we also had heard, for example, that AT&T's share of peering traffic of some other carriers was growing, which confirmed our view that AT&T was growing faster than the industry average. Thus this evidence was also consistent with our 2x estimate.
Roberts: It is true that some of the carriers we measured were growing at 2 per year but others were growing at 8 per year. The total traffic is the only sound measure since we all know that the sales capability and network quality of ISP's very considerably and some surge ahead while others slow down.
Odlyzko: Why would Larry Roberts obtain a higher estimate of Internet traffic growth rate? I do not know, since his data is not available for inspection and analysis.
Roberts: All the data is presented. The fact the names are not given is not a lack of data.
Odlyzko: Possible sources of the discrepancy could include the fact that he measures traffic using the 95-th percentile method, and not in terms of bytes. (A major reason Kerry Coffman and I used byte volumes is that those are pretty unambiguous, and also their growth rates show great regularity.)
Roberts: I have found that the 95% peak rate for trunks is totally
consistant with the Terabytes/month for the same speed trunk. It does vary somewhat by trunk speed. However, this difference would work in the other direction, since the OC-192 trunks are more efficient than lower speed pipes.
Odlyzko: There could also be problems with the quality of data that Larry Roberts obtained from carriers. It could be that he is measuring capacity growth more than traffic growth (and it has generally been true that capacity has grown faster than traffic, as
Kerry Coffman and I have pointed out many times).
Roberts: Actually, total Internet capacity and actual used capacity are growing exactly together. Some ISP's do grow capacity faster and we correct using their utilization. But others are running over 40% utilization and the variations cancel out. But, to be clear, we collect exact utilization by trunk for all trunks and compute the actual used capacity. There could be small errors in some cases if the ISP did not measure their utilization correctly, but Cisco and Juniper have very good programs which measure utilization all the time and all ISP's collect this data since it tells them when to order new trunks. Thus, any errors are very small and the net total result would not be affected due to the mass of data involved.
Odlyzko: My papers with Kerry contain some amusing (or appalling, depending on one's view) examples where people in positions of authority quoted figures that turned out to be wrong.
Roberts: This is why I insisted in collecting 3 samples before
believing the data. This let me check the consistency and accuracy of the source and when we suspected a problem we have gone back and insisted on hard data rather than one persons bragging.
Odlyzko: A major reason I am concerned about the quality of the data and analysis that Larry Roberts relied on is that there seem to be numerous small problems evident in his PowerPoint presentation. Let me just cite a few points. They may seem trivial, but they are bothersome.
(A) On slide #3, it is claimed that this is the first thorough study
of Internet traffic since 1996, when the NSFNET statistics stopped becoming available. Well, that is not correct, as is easy to check. NSFNET statistics collection ceased in April 1995, when the NSFNET was phased out. Even the late-1994 and early-1995 NSFNET statistics cannot be regarded as giving a representative view of Internet growth, as traffic started shifting away from NSFNET by the end of 1994. Thus we have not had a thorough study of Internet traffic since 1994. (The NSFNET traffic statistics and reports are easily available on the Web, at <http://www.merit.edu/merit/archive/nsfnet/statistics/history.bytes>.)
Roberts: I have had the NSF data for many years and I collected the peak bandwidth not bytes. In the past I had projected the future as well. It appears that the 1996 data point was a projection, not original data. However, if this data point were removed, there would be no change in my findings.
Odlyzko: (B) On slide #9, it is claimed that in other countries,
Internet traffic is growing 2.8x per year, and that this was the
historical (pre-2000) growth rate in the US. What is the source of this claim? Has Larry Roberts obtained data from foreign carriers? (Their growth rates vary widely, from about 2x per year for Australia to about 4x for international Internet bandwidth for China.) Also, what evidence is there that 2.8x was the annual growth rate in the US before 2000? The NSFNET statistics cited above show a remarkably regular 2x annual growth rate in the early 1990s.
Roberts: The carriers in many countries Ii visited this year have
given me their growth rates and most all have been about 2.8 per
year. This is quoted, not as a measurement, but as what it is,
reports from individual carriers.
My figure of 2.8 for the US is computed from the last NSF data and the first data point I have measured (April 2000). The traffic would have to have grown at 2.8 per year over the years in-between to span the gap. Since there have been no measurements, one can only compute the average growth rate.
Odlyzko: (C) On slide #11, we have a graph showing growth in
Internet traffic, starting in 1970. It appears to show that some
measure of traffic (it is not clear whether it is the 95-th
percentile or some other one) grew from about 20 bps in 1970 to about 300 Mbps in 1995. Well, growing by a factor of 15 million (300 Mbps divided by 20 bps) in 25 years corresponds to a growth rate of 1.94x per year, directly contradicting several slides where it is claimed that the growth rate historically was about 2.8x per year. Note that a 2.8 growth rate, combined with traffic at a level of 300 Mbps in 1995 could only come from a traffic of just 0.002 bps in 1970, an implausibly low figure.
Roberts: I never said the traffic grew at 2.8 per year from the start. It clearly did not. At the start is was around 1.3 per year and around 1988 exceeded 2.8. The 2.8 figure was for 1996 to 2000.
Odlyzko: (D) On slide #13, it is estimated that prices are dropping in half each year, while traffic is growing 4x, which implies revenues should be growing 2x per year. Yet we do not see such rapid growth in revenues. Generally, carriers' IP revenues from business customers seem to be growing at 30-50% per year. Furthermore, since prices are not in general dropping in half annually (with the exception of some major routes where there is a lot of fiber and heavy competition), this claim seems very questionable.
Roberts: I made no claims about revenue or price in the data reporting section of my report. These comments were qualified as personal comments, not reported data. However, to support my comments, I would point out the following: Most ISP's do not report IP revenue alone and thus there is no way to look at growth of revenue. I wonder where Andrew got his 50% number, AT&T? But since traffic has grown at 2.8 per year for most all years where reports might be available, with prices falling at 2/year, this would produce Andrew's 40% per year revenue growth into the start of 2000. We have been told by several ISP's that the price/bit is falling by 2 per year, mostly due to the lower cost/bit of T3 host lines. Although I have been researching revenue, I am not yet ready to publish anything more definitive.
In general it would seem that this all misses the point. I went out and measured all the carriers and determined the actual traffic in the Internet. No one else ever has. There is so little data available publicly that I gave up on projecting anything from such poor public data. In my mind, that is like examining an elephant by looking at its toenails. Since the names on this data cannot be released, I published the data points and the totals. For many people this has
been valuable.
--
========================================================
The COOK Report on Internet, 431 Greenway Ave, Ewing, NJ 08618 USA
(609) 882-2572 (phone & fax) cook@cookreport.com
Go to cookreport.com for info on new features, updated mission
statement, contents of Dec 2001 issue, subscription info and prices at
cookreport.com -- summary of content for
nearly 10 years at cookreport.com |
