Rapid i/o wants to do what HDS did inside its new storage..embedded switching...
techweb.com
September 18, 2000, Issue: 1131
Section: System Design -- Focus: Serial Interconnects
--------------------------------------------------------------------------------
RapidIO raises PCI stature in the box
Richard Jaenicke, Director of Product Marketing, Mercury Computer Systems Inc., Chelmsford, Mass., www.mc.com
The RapidIO switched-fabric interconnect technology announced earlier this year is designed to provide high bandwidth and low latency for intrasystem communications in embedded systems. Generally this means chip-to-chip and board-to-board transfer of data and control information within a chassis. One of the largest application areas for RapidIO is the communication fabric inside networking equipment, both at the branch office, for data and control information, and in public-scale networking on the control side only.
To enable the quick insertion of RapidIO technology, developers designed it to leverage legacy hardware and software. For example, the control plane of many pieces of networking equipment uses the PCI protocol with control processors and other functions attached to it. This environment helped to drive the design of RapidIO as a microprocessor bus and to interface seamlessly to PCI. As a result little need exists to replace PCI interface chips with RapidIO chips, unless the PCI interface is the limitation on the bandwidth. The path between RapidIO and PCI will be invisible to software.
The motivation for the development of RapidIO came from the mismatch introduced by the increase in processor speeds over the past several years and the bandwidth available to feed data to the processor to keep it busy. This disparity also applies to the speed increases of the PCI bus. In the same time that microprocessor clocks have accelerated from under 100 MHz to well above 600 MHz, PCI bandwidth has grown from 33 MHz to 66 MHz and then 133 MHz with PCI-X.
Meanwhile, PCI buses have grown progressively restrictive with respect to the number of devices that can be connected. As with any bus, PCI imposes electrical and physical constraints to ensure signal integrity. These constraints, however, limit PCI's ability to efficiently scale to support large numbers of high-speed processors and peripherals. This problem is exacerbated by the drive to increase bus frequency. Where the original 33-MHz PCI bus could support 10 loads, today's 133-MHz PCI-X standard allows only two devices per bus segment. At this top speed, PCI-X is essentially a high-speed, point-to-point connection.
In the past, PCI-to-PCI (P2P) bridges helped to overcome the limitations on scalability of a single PCI bus segment. Bridging allows up to 256 PCI bus segments to be connected together, but any transaction traversing multiple segments blocks those segments from use by any other device for the duration of the transaction. Bridging does enable scalability, unless there are only two devices per segment, as there is with 133-MHz PCI-X. Because the bridge counts as one device on each of the PCI segments it connects, adding bridges in that case does not permit any more devices to be added.
The obvious solution for scalability is to connect PCI bus segments using a switching fabric instead of a bus hierarchy. A switching fabric is an interconnection architecture that uses multiple stages of switches to route transactions between an initiator and a target. One benefit of switching fabrics is that each connection is a point-to-point link. This inherently provides better electrical characteristics, allowing higher frequencies that result in lower latency and greater throughput than bus architectures. The use of multistage switching also allows flexibility and scalability with regard to the size and topology of the interconnect. Examples of prevalent switching-fabric standards are switched Ethernet at the LAN level and Raceway inside high-end embedded system.
PCI can be combined with switched fabrics in a number of different ways to create scalable processor systems. One way to extend PCI with switched-fabric communication is to add a high-speed auxiliary communication network independent of the PCI bus. Such a design uses the PCI bus for basic control information and low-bandwidth I/O. High-bandwidth communication passes though the switched fabric with both high speed and low latency. As processing requirements of the application grow, more processors are added that interface to both the PCI bus and the switched fabric. On the PCI bus, this represents one more mouth to feed from the same bus bandwidth. The switched fabric adds another point-to-point connection for each additional processing node, thereby scaling bandwidth with processing.
This model provides a mechanism to add new technology such as RapidIO into existing architectures while providing compatibility with existing plug-in modules. By putting the interface to RapidIO on a separate connector, only new modules requiring the higher bandwidth of a direction connection to the switched fabric need to include any RapidIO circuitry. For example, in branch-level networking equipment, the existing cards would continue to interface with the PCI bus.
The data communications traffic for the new boards would be transmitted through the RapidIO switched fabric, with a single bridge from RapidIO to PCI providing a data path to the older cards. The control path for both sets of cards could be on the PCI bus.
Virtual PCI
In a more-integrated model, the switched fabric forms a virtual PCI bus for high-speed, low-latency concurrent communications between PCI bus segments. In this model, a PCI-to-switching-fabric interface must forward transactions from one PCI segment to another. The fabric interface translates PCI addressed to switch routes and fabric addresses, and then back again to the original PCI address at the destination PCI bus segment.
Virtual PCI switching fabrics must provide a software-transparent interface to the switching fabric as well as addressing all the issues of PCI bridging, such as handling PCI interrupts. Specifically, the only software that requires knowledge that there is a switching fabric present should be limited to the Power-on Self Test (Post) code in the BIOS and the OS.
The only software that needs knowledge of the internals of the switching fabric should be loadable from ROM by the Post code. Unmodified driver and application software then can logically treat all PCI devices on PC segments connected by the switched fabric as if they were bridged using the existing bridging specification.Examples of such virtual PCI switching fabrics have been around a while (See Aug. 10, 1998, page 80).
One example is the ANSI standard Raceway Interlink (ANSI/VITA 5-1994), which provides for transparent PCI switchwing when used in conjunction with off- the-shelf PCI-to-Raceway bridge chips. As the developer of Raceway technology and one of the developers of RapidIO technology, Mercury Computer Systems made sure that all the protocol features and architecture needed to implement a virtual PCI switching fabric are available in RapidIO.
RapidIO is designed for compatibility with integrated communications processors, host processors and networking digital signal processors. To software, RapidIO looks like a traditional microprocessor bus. Hardware implementations can thus hide things like discovery and error management. Features such as those help protect users' legacy software investment.
As a switching fabric, RapidIO provides scalable point-to-point interconnects with the following features:
- High Bandwidth: As processor technology advances at Moore's Law rates, communications performance needs to catch up. RapidIO provides the bandwidth to feed the fastest processors, with clock rates targeted from 250 MHz to more than 1 GHz. Data paths are 8 bits or 16 bits wide, and data is sampled on both edges of the clock. The resulting data rates scale from 1 Gbyte/second per full-duplex port with an 8-bit interface to 8 Gbytes/s with a 16-bit interface. The low-voltage differential signaling technology currently defined for RapidIO's physical layer will likely reach multiple GHz in the future, and RapidIO can be defined to use other physical layers as new technology rolls in.
- Low Latency: Connecting processors to each other and to memory within a system requires low latency. RapidIO addresses this requirement through the use of small headers and the absence of a software protocol stack.
- Determinism: Multiple priority levels in RapidIO promote determinism by ensuring the most important transactions always get through. Each packet is assigned a transaction priority, with each priority associated with one of three transaction flows. Higher-priority flows take precedence over lower-priority flows, and RapidIO moves them before continuing with less-critical traffic. Within flows, all transactions are completed on a first-come, first-served basis.
- Low Cost: RapidIO's low cost results from the small number of transistors required for each end point. RapidIO end points fit easily into a field programmable gate array (FPGA) with plenty of room left over for application-specific logic. One implementation consumes about 20,000 gates in a modern FPGA and provides raw bandwidth up to 620 Mbytes/second simultaneously in each direction. Leading FPGA manufacturers are expected to add RapidIO end points to their libraries.
Similarly, peripheral chip designers may choose to embed RapidIO end points in a small corner of updates to chips that are already on the market. For example, chips that today bridge PCI into a proprietary microprocessor bus will likely evolve to bridge PCI into RapidIO. Bridge chips are already evolving to include switching features. One example is Tundra's PowerSpan chip, which interfaces a PowerPC processor to two PCI buses with an internal switch. Some RapidIO interface products are likely to include a small embedded RapidIO switch as well. And where separate switches prove necessary, switches with multiple ports are likely to sell at prices similar to today's PCI-to-PCI bridging chips.
- Software Transparency: RapidIO has its origins in the microprocessor bus, and its direct memory-mapped address model is very easy to use. This means that software written for today's buses will continue to work with RapidIO. RapidIO supports multiple programming models, including Numa with physical addresses, ccNuma and message passing. This permits simultaneous, distributed I/O processing and general-purpose multiprocessing within a single system.
Direct memory-mapped
RapidIO is well-suited for use as a PCI switching fabric because it uses a direct memory-mapped model and is intended for use inside a system. The direct memory-mapped interface provides a straightforward translation of most PCI transactions, as opposed to "heavy" message-passing protocols optimized for intersystem communication. Its small size means that a RapidIO interface and a PCI interface can fit in a small corner of a modern FPGA, keeping RapidIO a cost-effective solution in the PCI space. It offers particular advantages in PCI-X systems, where the number of devices that can be connected to the bus itself is very limited. The 1-Gbyte/s minimum bandwidth for an 8-bit RapidIO port is a reasonable match for the 1-Gbyte/s raw bandwidth from a 133-MHz PCI-X interface, while providing scalability to hundreds of PCI-X devices.
Switched fabrics are already part of enterprise solutions and there is more to come. Soon the bandwidth demand will drive switched fabrics to cover communication between chips on a board, boards in a system, systems in a cluster and clusters in the enterprise.
Although RapidIO is well-suited for connecting processors and memory in desktop and server systems as well as in embedded systems, there are other solutions targeted directly at these nonembedded spaces. For example, the first use of RapidIO in a server could well be as a scalable I/O backbone that connects multiple PCI-X cards to a separate host processor and memory subsystem. Eventually, the broad acceptance of RapidIO and ultrahigh volumes of the embedded market could make RapidIO a compelling choice for all the switched-fabric communication inside of a desktop or server system. A RapidIO-to-Infiniband interface (as a host channel adapter) would be used to connect to other systems.
RAID systems
A second location for RapidIO in the Enterprise is in RAID systems. Although the connection from the server to the RAID system will likely be a switched fabric, like switched Fibre Channel or Infiniband, the RAID system needs internal switching as well. All of RapidIO's attributes intended for embedded applications apply here, too.
The RapidIO specification is speeding along toward becoming an open standard and will be owned by the RapidIO Trade Association. In addition to Mercury Computers Systems and Motorola, the original developers of the RapidIO technology, the steering committee includes Alcatel, Cisco, EMC, Ericsson, Lucent Technologies and Nortel Networks. Many vendors of PCI interfaces, FPGAs and switches, including Altera, Galileo Technology, PLX Technology, RedSwitch, Tundra Semiconductor and Xylinx, have also joined the RapidIO Trade Association. Lucent Technologies was the first to announce a product that is compatible with RapidIO, and others are expected to follow suit. |
