is.pennwellnet.com
SAN fabrics: Ethernet, Fibre Channel, InfiniBand
Fibre Channel is the only option for storage area networks today, but
Ethernet and InfiniBand are on the horizon.
By Bill Lynn
In the fast-paced world of e-commerce and the Web, there are two facts
that seem to remain constant. First, the amount of data is always
growing. Second, there can never be an interruption in the access to
data.
These facts are forcing a shift in the way systems are being viewed and
designed. Fading are the days where the server is the heart of the IT
environment and
everything else is a peripheral. In the future, the data (i.e., storage) will
be viewed as the heart of the IT environment. Key to this shift is storage
area network (SAN) technology.
SANs allow block-level direct access to storage by multiple servers. This is
different than network-attached storage (NAS), which operates at the file
level. The de-coupling of servers from storage has many advantages.
First, SANs allow storage and servers to be scaled independently of each
other. In other words, a SAN allows additional storage or servers to be
brought online without disturbing the rest of the systems in the
environment, or causing an interruption in the access to data. The ability
of storage to be accessed by multiple servers allows for greater fault
tolerance. If a server fails in a SAN, the ability to access the server's
storage is not lost; another server can assume the role of the failed
server. This ability is key to server clustering.
Click here to enlarge image
Fibre Channel is currently the only option for building SAN fabrics, but in
the future it may be possible to implement SANs using Ethernet and/or
InfiniBand.
Fibre Channel
Fibre Channel is a set of standards developed by the American National
Standards Institute (ANSI). Development began in 1988 as an extension of
the work being done on the Intelligent Peripheral Interface (IPI) Enhanced
Physical standard. Fibre Channel is a high-performance, full-duplex
interface that supports multiple topologies, physical interconnects, and
protocols. Devices running at 1Gbps are currently shipping in volume, and
a variety of vendors have begun sampling 2Gbps devices. Future interface
rates could include 4Gbps to support storage and 12Gbps (10Gbps after
encoding) to support fabrics.
The Fibre Channel architecture is based on layers, called levels, that have
the following definitions:
FC-0 defines the physical portions of Fibre Channel, including media types,
connectors, and electrical and optical characteristics needed to connect
ports.
FC-1 defines the transmission protocol, including the 8B/10B encoding,
order of word transmission, and error detection.
FC-2 defines the signaling and framing protocols, including frame layout,
frame header content, and the rules of use. It also contains independent
protocols such as login.
FC-3 defines common services that may be available across multiple ports
in a node.
FC-4 defines the mapping between the lower levels of Fibre Channel and
the upper-level command sets such as SCSI and IP. Each command set
mapping is defined in a separate specification.
Fibre Channel supports three different connection topologies:
point-to-point, arbitrated loop, and switched fabric. In a point-to-point
topology, two devices are connected together with a single Fibre Channel
link. The link can be a copper cable up to 50 meters in length, or an
optical fiber link that allows cable distances up to several kilometers.
Because a point-to-point topology involves only two devices, it is not a
suitable topology for SANs.
Click here to enlarge image
In an arbitrated-loop topology, devices are connected in a loop. In this
case, the receiver of a node is connected to the transmitter of the
previous node in the loop, and the transmitter of the node is connected to
the receiver of the next node in the loop, and so on. An arbitrated loop
can support up to 127 devices on a single loop, and the bandwidth is
shared among all of the devices.
The arbitrated-loop topology was originally developed as a low-cost way
to attach storage devices to Fibre Channel, and this topology still makes
up the bulk of Fibre Channel implementations today. One problem with an
arbitrated loop, from a SAN point of view, is that adding or removing
devices involves opening up the loop. This causes all traffic to stop on the
loop, and the loop has to be reinitialized. Some of these problems can be
circumvented through the use of Link Resiliency Circuits (LRCs) or Fibre
Channel hubs that include LRCs. While there are many SANs already built
using an arbitrated-loop topology, it is generally not an optimal topology
for SANs.
The true power of a SAN is realized using a switched-fabric topology. In a
switched fabric, multiple devices are connected together through a switch
or a series of switches. This topology allows for any-to-any connections,
where each connection has the full bandwidth available. Switched fabrics
can be very large, with millions of devices attached to the fabric. Fabrics
also allow for the addition and removal of devices without interruption to
the fabric, and also allow for the mixing of different speed devices. Figure
1 presents an example of a simple SAN that is connected using a Fibre
Channel switched fabric topology.
Click here to enlarge image
Even though Fibre Channel is the dominant SAN interconnect, some
problems remain. Interoperability at the device level is for the most part no
longer an issue, but interoperability between switches is still a potential
problem.
Another major issue is in the area of SAN management. Currently, a
majority of SAN management solutions require a separate Ethernet
connection to pass management commands. This is referred to as
out-of-band management. Solutions that support IP over Fibre Channel, or
in-band management, just recently began shipping. The Fibre Channel
Industry Association (FCIA) and the Storage Networking Industry
Association (SNIA) have formed several working groups to address these
issues.
Ethernet
Ethernet is the dominant networking technology. Work on Ethernet began
in 1980, and in 1983 the Institute of Electrical and Electronic Engineers
(IEEE) approved the IEEE 802.3 standard.
Ethernet follows a hierarchy that extends from the physical layer up
through the application layer. The reference for this hierarchy is the
seven-layer Open System Interconnection (OSI) model. The layers are
defined as follows:
Layer 1 is the physical layer, which defines the transport, including media
types, connectors, and electrical and optical characteristics.
Layer 2 is the data link layer, which defines the access method, such as
Ethernet or Token Ring.
Layer 3 is the network layer, which defines the routing protocols such as
IP or IPX.
Layer 4 is the transport layer, which defines transmission control
protocols such as TCP or UDP.
Layer 5 is the session layer, which defines the end-to-end session
control.
Layer 6 is the presentation layer, which defines application-specific data
formatting.
Layer 7 is the application layer, which includes e-mail, file transfers, etc.
In the past, Ethernet has not been considered as a SAN interconnect,
primarily because it was too slow and did not support a block-level storage
protocol. However, with 1Gbps switched-fabric Ethernet networks starting
to ship in volume, and 10Gbps speeds on the roadmap, speed is no longer
a major issue.
The main issue is developing a block-level storage protocol for Ethernet.
There are several efforts under way to develop a suitable block-level
storage protocol for Ethernet. For example, Cisco and IBM, as well as
Adaptec, have submitted proposals based on using the SCSI protocol over
Ethernet (see InfoStor, June, p. 1). A working group within the Internet
Engineering Task Force (IETF) has been formed to develop a method to
encapsulate the SCSI protocol, and a standard is expected within a year.
Ethernet has many advantages similar to Fibre Channel, including high
speed, support of a switched-fabric topology, long and inexpensive cables,
and a very large address space. The fact that Ethernet is ubiquitous in the
IT environment provides several other advantages, such as widespread
interoperability, a large set of management tools, and economies of scale.
A SAN based on Ethernet would basically look the same as the Fibre
Channel SAN shown in Figure 1, with the exception that the switch would
be an Ethernet switch.
The main near-term problem with Ethernet is the lack of a standardized
storage protocol. Another challenge is the fact that the transport
protocol, TCP, requires a large amount of processing by the CPU. The large
CPU overhead is not suitable for storage traffic, and has to be dealt with
by either off-loading TCP processing to an intelligent controller or using a
lighter-weight transport protocol.
InfiniBand
InfiniBand is the result of the merging of the Next Generation I/O (NGIO)
architecture and the Future I/O (FIO) effort. InfiniBand represents an
industry-wide effort to develop a replacement for the PCI bus. Version 1 of
the specification was due this month.
InfiniBand represents a significant change in server architecture. Figure 2
illustrates the architectural model.
In InfiniBand, the memory controller is connected to a host channel
adapter (HCA). The HCA is connected using InfiniBand links through a
switch, or a series of switches, to target channel adapters (TCAs). TCAs
are then used to interface to other forms of I/O, such as parallel SCSI,
Ethernet, or Fibre Channel. The TCA could also be the front end for an
external RAID subsystem.
There can be multiple HCAs and TCAs within an InfiniBand subnet. Subnets
are joined together through routers. InfiniBand is based on a 2.5Gbps
connection, and the links can be 1, 4, or 12 connections wide. Each link
can support up to 15 Virtual Lanes, which pass messages between queue
pairs. A queue pair consists of a send queue and a receive queue.
InfiniBand was designed to support storage, networking, and
inter-processor communications, and draws heavily from Intel's Virtual
Interface (VI) architecture. As such, InfiniBand is well suited as a SAN
interconnect. Actually, InfiniBand goes beyond SANs in that it is designed
to be an interconnect for system area networks, which allow for large,
high-performance clustered systems.
Even though InfiniBand has many potential advantages over current I/O
technologies, it is a major industry undertaking that involves sweeping
changes to server architectures. As with any new architecture, there will
be many issues that arise during the course of developing systems, and
many speed bumps along the way. Estimates vary, but it may be two to
three years until InfiniBand products begin shipping in volume.
Conclusion
Today, Fibre Channel is the only viable interconnect for SANs. Fibre
Channel is well suited for large IT environments where cost and complexity
are not an issue.
As SAN technology migrates down to smaller environments, storage over
Ethernet may become an attractive alternative. Ethernet is well
understood by IT personnel, and it provides the advantages of a single
network/storage infrastructure. And, due to the sheer size of the Ethernet
market, storage over Ethernet will likely have a significant price advantage
over Fibre Channel.
In time, server architectures will migrate to InfiniBand. By that time, SAN
technology will be fairly pervasive, with a large installed base of Fibre
Channel and Ethernet SANs. Figure 3 illustrates the relative positions of
the different I/O technologies in relation to system complexity.
Each of the I/O technologies fills a place in the SAN market, and they are
for the most part complementary. Each technology fills a different set of
needs, so that no one technology can satisfy the entire market.
Bill Lynn is marketing manager for Advanced Technology Strategy within
the Storage Networking Solutions division of Adaptec Inc.
(www.adaptec.com) in Milpitas, CA. He participates in the InfiniBand TWG
and AWG, FCIA, T11, and SNIA organizations |
