Marc Farley on InfiniBand
[Farley will be the keynote speaker at the upcoming Storage Networking World Spring 2000 event, presented by Computerworld and SNIA in Palm Desert, Calif. on April 17-19. Anyone going? storagenetworkingworld.com ]
This article purviews the FIO and NGIO camps and how their
reconciliation might play out within InfiniBand (SIO). It is
a great overview of the new architecture's promise.
performancecomputing.com
In August 1999, a rare event for
the computer industry occurred
when the two major I/O bus gangs
agreed to cease hostilities and
work together on the next major
system I/O connection
architecture. The NGIO (next
generation I/O) group, led by Intel with backing from industry
heavyweights Sun Microsystems Inc. and Dell Computer Corp.
(among others), agreed to merge their technology initiative with the
Compaq-led FIO (future I/O) group supported by Hewlett-Packard
Co., IBM Corp., and others.
At stake is one of the most influential and pervasive technologies,
one many believe will play a pivotal role in the speed of future UNIX
and PC systems. While the knuckle-down negotiations were no
doubt intriguing, what's even more interesting is what this new
technology is going to mean to the rest of us who are going to
purchase, install, manage, and use it.
In this article, we will examine the likely directions this new
technology will take, and its potential effects on clustered
computing, storage I/O, networking, and distributed data.
From Parallel PCI Bus to Serial I/O Channel
Against the backdrop of increasing application requirements for
faster, more reliable computing platforms that support
terabyte-sized data storage, the current system parallel I/O buses
are becoming obsolete. Figure 1 (p. 32) summarizes the
data-transfer capabilities of some of the parallel I/O buses currently
used in workstations and PCs.
Although the maximum transfer rates of SBus and PCI are fast
enough for today's high-throughput applications, the parallel design
of these buses hinders their future refinement. A specification
called PCIx has been drafted, which enables a single slot to
function at 133MHz across an eight-byte data bus for a theoretical
transfer rate of 1GB/sec.
But PCIx, which is backward-compatible with existing PCI cards,
does not provide balanced throughput and is otherwise saddled
with the same restrictions that existing PCI implementations have.
Hence the interest in defining a new I/O connection architecture.
This architecture does not have to be faster than SBus and PCI
initially, but it has to have the potential to increase its performance
by ten times or more in the next four years.
The new I/O connection architecture no longer includes a
connecting bus with 90+ pins. Instead, it is a serial-transmission
channel expected to use four-conductor copper or fiber-optic
cables. Parallel I/O bus lengths are restricted from a few inches to
a couple feet. Serial channels can extend over longer distances,
ranging from 15 meters to 6 miles. Parallel buses are also
restricted in the number of system board slots allowed for
connecting I/O adapters. Serial channels have no slots and are
limited more by protocol design than by physical signal-transfer
capabilities. Parallel buses usually carry power signals to the
adapters that plug into them. Serial channel cables do not conduct
power. This can be good and bad: supplying power through the
bus reduces the number of power supplies and cords needed, but
it also forces system manufacturers to provide power supplies
capable of supporting fully populated systems, even if those
systems will never need the additional power.
Most importantly, parallel I/O buses are a shared resource that
implements some kind of prioritization scheme and interrupt
processing that determines which controller is transferring data on
the bus. In contrast, serial channel cables extend directly from the
system without needing intermediary adapters, or interrupt
processing, to establish communications between entities on the
channel.
The integration of network switching technology in the channel
allows simultaneous multiple data transfers or commands in the
channel. In other words, the aggregate transfer rate of a serial I/O
channel can be several multiples of the single-link transfer rate.
Even though the link speed may be the same as a parallel I/O bus,
the serial channel is capable of much larger aggregate throughput.
In addition, parallel buses usually do not have separate send and
receive lines, as serial channels do. Therefore, it is theoretically
possible to double the bandwidth across any single link.
Serial Life
In the big scheme of things, the switch to a serial I/O channel
implies significant changes to many aspects of the system. Some
of the results of switching to a serial I/O architecture will include:
the implementation of external, non-shared, non-blocking
switched connections
lower latency communications between multiple channel
entities, particularly between systems in a cluster
dynamic system configuration and hot swapping
virtual controllers implemented in software, eliminating many
host adapters
smaller system dimensions due to the elimination of host
adapters and the reduction in power and cooling
requirements
new memory and memory controllers for connecting to the
serial channel
an increase in the number of host-based storage and
data-management applications
the blurring of the distinction between I/O and networking
Switching Switches
Something very interesting happens when you change from a
parallel I/O bus to a serial I/O channel: all the stuff that was
previously done electronically over multiple bus lines now has to be
done using a communications protocol.
The nature of the switch implementation, particularly the selection
of protocols, is supposedly one of the major differences between
the FIO and NGIO architectures. FIO uses open-standard protocols
and routing schemes, whereas NGIO implements a closed-switch
architecture that used proprietary protocols.
Both plans have merit and some of the differences between them
are surprising. Usually an open specification, such as the one FIO
proposes, is preferable for almost everybody involved because it
brings down the cost and price of components and products in
addition to providing broad interoperability. However, broad
interoperability is a matter of degree, and depends on whether or
not the specification is narrow and deep enough, and how well the
specifications are implemented. For example, the Fibre Channel
switch specification for exchanging simple name service
information between switches did not follow this rule-it wasn't
specified in enough detail to allow broader plug-and-play
compatibility among Fibre Channel switch vendors.
FIO risks having these interoperability problems, which could
create much bigger problems in the new serial I/O channel market.
With so many companies involved, looking for ways to create
unique implementations could be difficult, and might result in
delays in the specification or the release of a shallow specification
that had insufficient interoperability guarantees.
However, assuming the specification works, the next question is
how to safeguard exposed system resources, such as memory,
from poorly behaved applications and hackers. Many of us have
experienced problems with bad device drivers and the device
driver policy du jour some host adapter products exhibit in times of
trouble.
With a serial I/O architecture, however, the potential exposures are
much greater. Just as everything else connected to a network is a
potential security leak, a serial network I/O channel is no different.
Without adapters full of hardware providing a barrier to access for
incompetent or wayward coders, device-level hackers will have
unprecedented access to system internals. Obviously, this is a
technology direction that needs to take security very seriously.
The NGIO alternative, to use a proprietary switched architecture,
circumvents some of the problems discussed previously. Certainly,
if the specification is not open, it makes it much harder for the
wrong people to code to it. Realistically, however, a closed spec is
truly a hacker's paradise, and it wouldn't take long for groups of
motivated hackers to figure it out and publicize the funny stories
and interesting channel tricks they can perform to the digital
underground.
Where interoperability is concerned, a proprietary architecture
virtually guarantees it. That is until an opposing "open"
specification comes along to challenge it. About ten years ago,
IBM was fighting a losing battle to make its proprietary
MicroChannel architecture I/O bus the industry standard. Even
though Intel is a powerhouse, it is unlikely its proprietary I/O
"standard" could withstand an all-out assault from Cisco Systems
Inc., Compaq Computer Corp., Hewlett-Packard, IBM, and Lucent
Technologies, not to mention Advanced Micro Devices Inc., Cyrix
Corp., and hundreds of other companies wanting to get in on this
action. Without the same kind of vested interest Intel has in
protecting the license, Sun and Dell might not provide the kind of
support needed. A defection from either of them would be harmful.
Fighting for and losing a proprietary I/O channel specification
would be an extremely damaging endeavor, even to Intel.
For that reason, it is believed that serial I/O will include an open,
royalty-free, public protocol specification that allows all CPU,
system, I/O controller, and switching vendors to participate on more
or less equal footing. The difficult process of herding all these cows
together becomes the next spectator sport in serial I/O
entertainment.
How Much Data Can a Data Chip Chuck?
One of the other supposed differences between FIO and NGIO is
the architecture of the system bus interface. The I/O channel is for
connecting systems to storage, networks, and other systems. But it
is not intended to connect CPUs together in a single system, nor is
it intended to connect those CPUs to memory. That is the job of the
system bus.
Therefore, there needs to be some way for data transfers to cross
the boundary from the I/O channel to the system bus. In reality, this
means transferring data in and out of system memory from the
serial channel. There are several possible architectures for doing
this, and not surprisingly, FIO and NGIO do not see eye to eye on
this either.
NGIO's architecture revolved around the concept of a
memory/channel controller that was the intersection of the CPU
bus, memory, and the serial I/O channel. In essence, this
architecture establishes a tri-level structure with the controller and
system memory situated between the system bus and the I/O
channel as shown in Figure 3.
The major benefit of this architecture is the ability of I/O processes
to transfer data to and from system memory without interrupting the
CPUs. This lets the CPUs continue working on applications with
fewer I/O-driven context switches. As the amount of I/O traffic is
expected to increase with Gigabit Ethernet and Fibre Channel, it is
increasingly important to take CPUs out of the I/O-to-memory
transfer loop.
The obvious potential bottleneck in the NGIO architecture is the
memory/channel controller between the serial channel and the
system bus. The benefits mentioned earlier about multiple
simultaneous data transfers across the serial channel cannot be as
large if they don't include data transfers in and out of system
memory. That is, after all, the primary purpose for having an I/O
channel in the first place. Also, this single memory/channel
controller is a single source of failure. If this chip fails, so does the
rest of the system.
FIO, on the other hand, is designed to use multiple access points
onto the system memory bus. This obviously provides much better
scalability than a single connection, but it also introduces some
difficult problems. Instead of managing all data transfers to memory
through a single controller, the FIO scheme implements multiple
channel controllers. Multiple channel controllers reading and writing
to system memory all need to have a path to access that memory
without stepping on data already present.
It is assumed that the FIO plan includes a scheme for channel
controllers and system CPUs to function as multiprocessor peers
on the system memory bus to avoid system anarchy. The
technology used in the FIO channel controller would necessarily
have a great deal to say about how memory sharing is
accomplished, as well as how other multiprocessor
communications occur.
While large multiprocessor systems are common in the UNIX
world, they represent an elusive goal in the world of Intel-based
systems. This is something Intel undoubtedly wants to change, so it
is safe to assume Intel plans to deliver an architecture with the
processor scalability found in enterprise systems. Undoubtedly, the
I/O channel interface to this processor realm is already integrated
into that scheme and one can imagine a design where multiple
NGIO-style channel controllers can be deployed to work together to
access system memory.
In order for any such plans to work, Intel has to have control of the
system bus interface of the I/O channel controller. Intel seems to
believe that anything having to do with CPUs, such as the system
memory bus, belongs to the company and defends that technology
ferociously. It is unthinkable that a system maker like
Hewlett-Packard, IBM, or Sun would open up its system-memory
bus architecture to the drawn-out standards-development process,
only to find out that the architecture doesn't work and there is no
more competitive advantage for their processors. The inclusion of
an FIO-style channel controller in this domain could clearly
complicate Intel's technology and market road map.
So, following those arguments, the odds are very high (bet the
farm) that Intel and NGIO have prevailed in this part of the System
I/O agreement. One should expect that the SIO spec will include an
open-standard interface in the channel itself as proposed by FIO,
but will call for the NGIO channel-to-system-memory-interface
architecture. It's not a bad compromise; everyone gets what they
want and will be able to protect their business models. Now we'll
just have to see how accurate this assessment is after the SIO
organization unveils its plans.
The Protocol Picture
SIO could choose to use existing wire protocols or invent its own.
Using an existing protocol would certainly save development time,
as chip cores already exist for the current major protocols.
However, using an existing protocol could constrain channel
features and may not provide the kind of functionality required for
some of the channel's applications.
Conversely, the development of a new protocol requires a lot of
development work from many companies, and that always takes
more time than anticipated. However, a new protocol should be
able to meet all the requirements of an I/O channel for several
years. The last reason for choosing a new protocol falls into the
camp of solving a problem that doesn't exist yet-adapting the new
protocol to future requirements. It is difficult to predict what future
I/O channel requirements will be (other than louder and funnier), but
there is reason to believe that a new protocol developed with this
goal in mind could be more successful than one that was
developed years ago.
The four major system areas to be addressed by the serial I/O
channel are storage, network, cluster, and video I/O (including
multimedia).
The first three of these system areas have their own
network-protocol implementations. For example, storage I/O has
the Fibre Channel protocol (FCP) serial mapping based on
SCSI-3 specifications. If the System I/O channel used low-level
Fibre Channel layers for its native transport protocol, by extension
it would be fairly simple to extend the System I/O channel for
storage area networks (SANs). With Fibre Channel for System I/O,
all data network traffic needs to be converted to networking
protocols through bridges, routers, switches, or Ethernet controller
units connected to the system channel.
Of course, it would also be possible to use Gigabit Ethernet as the
native transport protocol. Gigabit Ethernet is fast enough and
supports the full I/O protocol suite including TCP, UDP, SNMP, and
others. In that case, access from the System I/O channel into
Ethernet networks would be technically simple, but would require
some intense security and firewall mechanisms to keep unwanted
IP network traffic off the system channel.
Certainly, other common network transport protocols could be used
as the transport for System I/O, including FDDI and ATM. In both
these cases, and with Gigabit Ethernet, it would be necessary to
use a bridge or controller to convert storage traffic from the System
I/O channel to a format for storage I/O traffic. Interestingly, although
it has not been available for very long, the FC/IP (Fibre Channel IP)
protocol could also be used as the System I/O's network protocol
implementation-allowing a single transport protocol (such as Fibre
Channel) to be used for both storage and network I/O across the
System I/O channel.
It is hard to imagine that the transport protocol selected for System
I/O would not support cluster communications as a native
capability. While cluster I/O has several protocols available, the
one most interesting for System I/O is the virtual interface
architecture (VIA) protocol developed initially by Compaq, Intel,
and Microsoft and now supported broadly within the industry. While
VIA could be used for many things, it is directly targeted at
low-latency cluster communications for such things as heartbeat
and distributed-locking applications. Work is currently underway in
the Fibre Channel community to complete the upper layer protocol
(ULP) mapping for VIA-over-Fibre Channel (FC/VI). Whether it is a
trend remains to be seen, but in September 1999, Troika Networks
announced the first host bus adapter that supports the three Fibre
Channel upper layer protocols mentioned above (FCP, FC/IP, and
FC/VI).
Figure 4 shows the relationship between three possible transport
protocol selections for the System I/O channel. A new protocol
could be used that allows cluster communications to occur. In that
case, a bridge or controller function would be needed to connect to
both data networks and storage networks. If Gigabit Ethernet were
chosen, it would be necessary for a bridge/controller to convert
traffic for storage I/O. If Fibre Channel is chosen, storage, network,
and cluster communications could all be carried over a single
transport. It is likely that a bridging function would be needed
between Fibre Channel and Ethernet, but notice that the IP
component would not have to be transformed.
Video I/O traffic seems to be the furthest behind with the RTP
(real-time transport protocol) being developed for applications
such as video conferencing. It will be interesting to see what
happens with video, as it has not seen a lot of attention as needing
requirements for network protocols. The System I/O channel is
likely to change that.
The chances are good that protocol development for System I/O
will continue to go on for many years as vendors find better ways to
use the bandwidth of the channel more effectively across multiple
applications in distributed data and computing environments.
These will be interesting times for the industry, the new standard
serial I/O channel will change the face of computing as much as
any other technology change in the last decade. This might seem
like an overstatement, but when one considers that it will radically
affect the way systems connect to networks, their storage and each
other, it's difficult to argue with the assertion. Companies
developing I/O adapters are likely to see the largest degree of
change. If you make your living selling controllers that fit into PCI
bus slots, the change to a zero-slot serial I/O channel is not very
welcome.
But at the end of the day, SIO means that all I/O activities will be
integrated into a network environment. Sun's marketing slogan
from several years ago, "the network is the computer" becomes
more applicable. By replacing the traditional parallel bus with a
serial network I/O channel, there is not much left inside the system
anymore that is not integrated with networking technology. This is a
powerful change, rich in possibilities, that will continue to spur
innovation.
Brian Berg contributed to this article.
Marc Farley is the vice president of marketing at San Castle
Technologies, a startup company making high-speed switching
products for storage networks. His latest book, Building Storage
Networks, was published recently by Osborne McGraw Hill. |
