5/98 Boardwatch must-read article by Rickard (Part I) [Good ASND reference]. "YET ANOTHER UNIQUE MOMENT IN TIME PEERING REDUX - BACK TO THE FUTURE AND THE ESSENTIALS OF A COMPETITIVE INTERNET"
boardwatch.com
Excerpt: "But behind the scenes things are not so dark. A number of new players have
come into play or are trying to. Large, but almost silent networks such as
AT&T and IBM are increasingly disenchanted by the status quo and the
emerging WorldCom/MCI new world order. Infrastructure companies such as
Qwest Communications, L3 Communications, and Williams
Telecommunications are also wanting to be players as well as virtually all of the
regional Bell operating companies. The cobbled together NAP architecture has
never appealed to these companies who are accustomed to providing
carrier-grade quality assurances to customers who do not understand why they
now can't when it comes to Internet services. Williams Telecommunications has
come out publicly to be among the "founders" of the BPP Group and both
Qwest and AT&T are rumored to be interested. Small backbones such as
Exodus Communications and Electric Lightwave have similarly signed on. And
Ascend Communications may have stolen a march on the other hardware
manufacturers by signing on as an early sponsor of the effort with technical
assistance and their Ascend GX550 25-100 Gbps ATM Switch.
One of the drivers behind all of this actually has not much to do with peering
per se. There are a host of new Quality of Service products waiting in the wings
that simply won't work across the existing NAP structure. Everyone is excited
about doing voice over IP, for example, but the existing level of quality of voice
across the Internet is unlikely to appeal to anyone but experimenters and very
strong power users. And there is no way for any one network to unilaterally fix
this. Provisions for these sorts of services have to be provisioned in the
interconnect space between networks or they just never can quite work.
In our estimation the Gaddis white paper presents one of the best blueprints
we've seen for a radical new peering policy for everyone across the network
that would probably work."
____________________________________________________________________
In the beginning, there was an Internet - a bold proposal to link 13
geographically, technically, and organizationally different networks into one
linked supernetwork spanning the continent at a breathtaking 56 Kbps using
Vinton Cerf's Transmission Control Protocol over Internet Protocol. It gave
remote access to programs and resources on some of the nation's largest
supercomputers. And it demonstrated that you could link very disparate local
area networks to do useful, if basic things such as convey electronic messages,
transfer files, and login remotely.
The National Science Foundation funded this link and a company titled
Advanced Network and Services (ANS) was created specifically to build it,
using resources from IBM and MCI. It worked, and it served as both a
symbolic and technical center for the unification of all computerdumb. It gained
sufficient critical mass to break the problem of e-mail "islands" then extant in the
commercial network arena, and gradually subsumed all networks everywhere.
At some point, it was no longer a demonstration, and much of the use of the
Internet had little to do with academia, or anything the National Science
Foundation was chartered to do. The obvious solution was to privatize it -
allow private commercial companies to operate it and sell access to it
commercially.
A huge brouhaha ensued - a melee of epic proportions. IBM and MCI had
essentially built the National Science Foundation backbone, and claimed to
have an identical "private" backbone, using the same rooms, same equipment,
and same technicians as the NSF backbone. They viewed themselves as the
natural inheritors of the Internet and pointed to their considerable investment in
developing technology for the NSFNet backbone as sufficient justification for
their inheritance.
A small band of protestors howled in rage that a government program funded
by tax dollars could be "given" to a private company such as IBM and/or MCI
- IBM being the devil incarnate at the time in the personage of Alan Weiss.
Rick Adams of Alternet, later renamed UUNET, Marty Shafkowitz and Bill
Schrader of Performance Systems International (PSI) and a few others made
sufficient ruckus that they too should be allowed to participate in the riches that
would flow from privatization.
A mailing list was formed - Commercial Privatization or COMM-PRIV to
discuss every conceivable variation of every conceivable conspiracy between
IBM and the NSF - with Gordon Cook, previously of the John Von Neumann
Center and destined to become something of the Rona Barrett gossip columnist
of the Internet fueling the debate with e-mail missives exceeding the length
restrictions of most of the mailer software of the day.
PSI and UUNET largely, though with participation from Sprint, formed the
Commercial Internet Exchange to demonstrate network-to-network exchange
of traffic, as well as to provide an ostensible shunt around the NSFnet
backbone to justify early commercial use of the Internet in the face of the
government Acceptable Use Policy. Along with the Metropolitan Area
Ethernet's developed by Metropolitan Fiber Systems (MFS), and the Federal
Information Exchanges (FIX) the concept of exchange points evolved.
In the end, the domain name and IP address assignment tasks were awarded
by contract to NSI, a routing arbiter database contract was issued to Merit,
and a series of Network Access Points or NAPs was designated. Originally
three, they were awarded by contract to Pacific Bell in San Francisco,
Ameritech in Chicago, and Sprint, which was known as the New York NAP
though located in Pennsauken, New Jersey. Before it was quite a done deal,
everyone seemed to agree that MAE-East in Washington, DC, should be the
fourth of the three NAPs.
A bit of a combo problem was left - peering and settlements. And after months
of wrangling, it was decided just not to decide it. Anyone who showed up at
any of the NAPS could peer with anyone they wanted to, not peer with anyone
they didn't want to, and work out any settlement issues they could as best they
could. The NSFnet backbone was shut down a year later in April of 1995. The
commercial Internet was go and that a basic problem of what it actually was or
how it worked was overlooked in the flush of incoming business and money.
PEERING AND SETTLEMENTS
Peering and settlements has proven almost unsolvable on the Internet. Though
unsolved, the two continue to be linked intrinsically. Peering is an agreement to
exchange traffic and route advertisements at a NAP. The traffic is a bit obvious.
If you have a user on one network and a web site on another, in order for the
user to receive the web page, Internet Protocol packets have to transit from the
one network to the other somewhere. In theory this would be at the NAPs. But
in leaving peering an optional item, not all networks exchange traffic with each
other. Traffic must often go through an intermediate network to get there.
The route advertisements are even more key. Routes are paths to a particular
Internet resource. If network A doesn't get route advertisements from network
B, then no one on network A will know how to get to a web site or resource
on network B. Indeed, they just won't appear to be reachable. As a result,
peering problems have long made the Internet much less ubiquitous than
perceived. Entire portions of the Internet wink in and wink out as seen from any
particular location - mostly due to peering disputes and glitches. Customers
have been put off with vague explanations of "problems" when they were
suddenly unable to see selected resources. The common knowledge that you
can reach any point on the Internet from any other point has, unfortunately,
never held true for more than a day or two at a time.
The settlements issue, is probably the real heart of all the problems on the
Internet today. In the voice telephone network, they simply opted on a simple
rule - caller pays. Whoever initiated the call, pays for a long-distance call. And
the originating network pays a fee to the termination network. Since calls run in
both directions, each network is credited with hundreds of transactions each
hour. On a monthly basis, the "balance" is calculated and if there is a disparity,
the balance of value is paid in cash as a "settlement."
Even in this simple instance, settlements have a basic overhead problem. Let me
spend $10 to determine that I owe you $100. You likewise spend $10 to
determine that you owe me $105. So five bucks changes hands, but we spent
$20 to get there. Fortunately, in the voice network, the charges were dwarfed
by what the consumer paid in per minute charges - so we'll pass this madness
on to him.
But even in the voice network, settlements are actually undergoing a lot of
review and examination right now as a perhaps inefficient concept. On the
Internet - it just doesn't work at all. First, none of the customers pay per minute
charges. Second, it's devilishly difficult to determine who initiated the call in a
technical sense. And the flow of value is almost incomprehensible. Does the
dial-up user who contacts a web site gain value because they can contact it? Or
does the web site gain value in getting a "hit" from a caller? Which way does the
"value" flow in this type of information exchange.
The basic unit of currency could be the number of customers. A network with
100,000 customers might consider access to its network by a network with
500 customers a real value to the smaller network. Now if the 100,000 were all
dial-up customers, and the 500 were all huge web sites such as Microsoft,
Netscape, Yahoo, and Excite, would this still be true? Is the network providing
access to these "resources" the one really providing the value? Or would the
first network be providing the second network with a valuable "audience" that
they should pay to access.
Traffic analysis only worsens the situation. A "caller" sends a single 1500 byte
packet requesting a page to a web server. The web server responds with an 80
KB graphic, seven 8 KB graphics, and a page of text. It is an asymmetric traffic
flow to the point of absurdity.
The constant jealousy over who is carrying whose water at any particular
moment led to a technical innovation termed "hot potato routing." In this,
individual packets are hosed off onto the terminating network at the earliest
opportunity based on the destination address of the packet and without regard
to any sort of routing efficiency beyond the fact that "this packet is headed for
you and I don't want to pay to carry it another mile." Why that packet is
headed that way varies from application to application, of course, and the entire
concept, based in greed and malice, makes about as much sense as charging
Santa Claus a penny for every packet sent on the Internet as a whole. But it
evolved in this manner. As a result, a page request might cross the entire
continent on Sprint's network, and the responding page itself transit the entire
continent over MCI's network - ALL of this entirely different from how a ping
or traceroute would show the same connection.
And it only gets worse. FTP sites, e-mail traffic, USENET news, with every
new application the concept of a flow of value becomes more absurd. The
analogy we've used for years, is that two identical twin brothers, each wearing
nice suits and the best of intentions, motivated by a million dollar bonus for each
of them if they could come to some agreement, could never work out a value
exchange system on the Internet. So is it any wonder that nearly 40 backbone
networks, with somewhat more complicated motivations, could fail to do so as
well? Everybody knows the other guy should pay something. But no one knows
what he should pay, why he should pay it, or who the other guy is precisely. It
just seems like everyone else is getting a free ride.
So settlements never were settled. And inescapably this leaked into peering.
Larger networks felt they were "enabling" smaller networks by peering with
them and exchanging traffic and routes. They were providing access to a larger
network for free. But it's the same basic moronic argument. Are 200 millionaire
customers of less value than 20,000 poor customers? Dial-up users versus web
sites? Business customers vs. consumers (complicated by the fact that the
business customers obviously WANT access to consumers, and versa visa). I
have more customers than you. Yes, but I have better customers - intrinsically
more valuable ones. And a third network has but a handful of customers -
including PointCast Network, Netscape, and Yahoo. Same leak springing from
a different hole.
What emerged was chaos. At four NAPs, some peered with others, and some
did not. UUNET, interestingly, was a very promiscuous peerer. For some time,
they would peer with anyone with a clue, and some who had no clue, and in
some cases helped people get a clue so they could peer. MCI was very
reluctant to let all these "lesser" networks peer with them at all. But if pressured
by their paying customers, they would. Sprint ran it both ways depending on
what period of time we are talking about and who was at the helm. These three
networks together comprised most of the traffic on the network. And in a kind
of ongoing competitive jealousy bordering on rage, the Internet teetered along
with a peering policy that was essentially no policy at all.
Peering is somewhat more complicated at a technical level than quite described
above. Giving another network the power to shoot routes into YOUR network
is the analog of a promiscuous commingling of body fluids. The network you
peer with has the power to totally WRECK your network - either through
malice, or more commonly, error. A couple of mistyped lines and neither one of
you are doing much Internet that day. With backbones arising almost as the
result of high school student science projects, this gets to chaos pretty quickly.
Further, as the number of networks increases, the cost of just managing ports at
NAPS to accomplish peering becomes non-trivial. If nothing else, your contact
list of people to call when something goes wrong becomes unmanageable, the
chances of something going wrong increases dramatically, and the time spent
trying to fix whatever went wrong rises in scale. It's just not true as a practical
matter that everyone with a 2400 bps modem that wants to dial into a NAP
and be a peer can do so.
Last year, UUNET announced an "end to free peering" and an end to the "free
ride" and sent dark e-mail messages to dozens of smaller networks advising
them they now had to be "customers" and pay for connections rather than peer
at NAPs. There was a lot of howling, but in the end, we can't actually find a
single case of anyone who lost peering. UUNET wasn't actually able to make it
stick. Same by now aging problem. Anyone cut off would cause angst from
UUNET customers. Was UUNET disconnecting the smaller networks from the
Internet, or was UUNET disconnecting ITSELF from the Internet? It was a bit
of a bluff and it largely failed. Made a nice Boardwatch cover of John
Sidgmore rolling a few barrels of ammonium nitrate up to MAE-East. That was
probably the biggest effect it had.
PERFORMANCE
Aside from these ongoing perceptual difficulties regarding peering and value, the
four original NAPS have emerged as major bottlenecks to Internet
performance. Traffic has expanded enormously, and getting it through the
NAPS themselves has proven to be a huge performance problem. With
Keynote Systems, we've run over 10 million discrete web page download
measurements crisscrossing the network in nearly every conceivable direction.
What has emerged is that most of the performance problems appear to be
exhibiting themselves in the interconnect space. Two networks can both work
admirably across their entire topography. But if you put a client on one and a
server on the other, it can look very bad very quickly if you try to time packets
crossing the chasm. The NAPS are the lion's share of this problem - they
inherently have two many packets in one room.
PRIVATE INTERCONNECT
This gets worse in solution than it ever was just as a problem. The larger
networks are choking down bandwidth into the NAPS, and instead working
out "private interconnect" solutions. If GTE and MCI are sprouting a lot of
traffic in Atlanta between each other, and all of that has to be backhauled to
Washington, DC, to exchange it in an already choked MAE-East NAP, it
makes economic sense to simply interconnect in Atlanta, swap the traffic
locally, and move on. As this grows, however, it contracts the "good old boys
club" to a handful of huge networks that develop this private interconnect
outside of the public NAP structure. One problem with this is that there is still
traffic through the NAPS to resources located on smaller networks, and
performance to those sites is egregiously poor. But over time, it is almost
inevitable that this private interconnect space will be seen as anti-competitive
and ultimately collusive. It intrinsically allows large players to develop
interconnect excluding new entrants and smaller players. It is certainly NOT
offered to all comers - or even based on specific criteria. There is only one legal
outcome possible, and it awaits really a single lawsuit to become an entirely
untenable situation. If you thought school busing was bad, you haven't seen
anything until you see an Internet redesign under court supervision.
WORLDCOM/MCI
The thing that has held the peering situation in the air to this point has been a
basic balance of power between three large and very competitive networks -
MCI, UUNET, and Sprint for some four years now. Mix in some smaller, but
historically venerable networks such as PSInet, BBN Planet (now GTE
Internetworking) and the commercial version of ANS, and you have a kind of
precarious balance that did afford smaller networks and new players an entr,e
into peering at the NAPS. It was time consuming and difficult to gain peering
and even large networks such as CompuServe had actually a tedious time
gaining peering at the NAPs initially. But it could be done.
The lingering theory revolves around the concept that one network could
become sufficiently dominant to change the game and essentially "steal" the
Internet by simply announcing radical change to peering policy. In the past year,
some interesting developments have occurred. WorldCom purchased MFS
Communications, GridNet, ANS, UUNET, and CompuServe. And in March,
shareholders in both companies approved a $37 billion merger between
WorldCom and MCI. WorldCom has acquired enough of the other backbone
companies to claim well over 50 percent of all connections to all smaller ISPs,
well over 50 percent of the total traffic, and probably close to 70 percent of all
business accounts. Complicating this is the fact that most of the other smaller
backbones with the exception of Sprint, PSINet and AT&T get their underlying
physical infrastructure from WorldCom. And finally, UUNET as described
earlier, had already made an abortive run at radically changing the peering
balance of power last summer - before having the critical mass to quite pull it
off. With the WorldCom/MCI merger, it is widely feared that they WOULD
now have the critical mass and would indeed be able to pull it off. Being
connected to the Internet increasingly looks analogous to being connected to
WorldCom. And being disconnected from WorldCom could look a lot like
being disconnected from the Internet.
In February, the European Union (EU) announced they were reviewing the
WorldCom/MCI merger and had some serious questions about Internet
competition. Within days, the U.S. Department of Justice announced they too
were having a closer look. And the FCC has also noted reservations about the
merger. It does not seem to be an issue of voice telephone service or long
distance telephone service. The entire concern centers on Internet dominance.
We've heard quite a bit of surmise that these are largely pro forma and that the
merger is due for inevitable approval. Our best information would lead us to
believe otherwise. It's probably in a great deal more trouble than commonly
believed. At this point, I would rate the chances of unfettered approval as
approximately nil. But if a way were found to approve it and ensure everyone
of a truly competitive Internet environment globally, it would eventually happen. |
