From Unix Insider,
Excellent summary of i/o history and the coming of infiniband.
sunworld.com
InfiniBand is coming
New connectivity technology is about to arrive, but what will it look like?
Summary
InfiniBand was recently released by the InfiniBand Trade Association to allow vendors to implement products based on it. Some vendors claim they'll have products by mid-2001, barely eight months away. In this month's Connectivity, Rawn Shah considers what that impending arrival means to the PC's future. (2,000 words)
--------------------------------------------------------------------------------
CONNECTIVITY
--------------------------------------------------------------------------------
By Rawn Shah
hen InfiniBand arrives in 2001 for servers and workstations, we'll be faced with the first of a new type of server. Being first isn't always best when it comes to implementing a new technology. You're working with new rules, figuring out how to administrate it, learning new requirements for compatibility with other products, and struggling with higher costs for the initial products.
Those problems aren't characteristic of InfiniBand itself, but of the product development and marketing process. Vendors that are first to market a solution often face higher production costs and, having little competition, they sometimes raise their list prices. At the same time, there's little to test connections against. Finally, the product may not be supported under existing management software products.
For InfiniBand to succeed, we need to understand what computers look like, how they work, and why we need this next step at all.
A history of buses
InfiniBand faces the initial challenges mentioned above, making it hard to predict when and where that new high-bandwidth interconnect will be most useful. Fortunately, we have two examples of new peripheral interconnect systems to compare it to: the EISA and Microchannel buses, and the PCI and VLB buses.
When the Intel 386 was first coming into wide use, IBM, HP, and other vendors decided that the 8.3 MHz, 16-bit ISA bus wasn't sufficient to run the high-performance applications of the day. The ISA vendors group and IBM both led a charge to develop a new 32-bit bus platform that would be both efficient and effective. The EISA standard was a doubling of the ISA system with twice the number of interconnects of ISA, and it introduced bus mastering to allow peripheral cards to communicate more readily. IBM's Microchannel was a better idea technologically, but as in the VHS-Betamax wars, EISA was easier to implement into products without costly licensing from IBM. In the end, even IBM gave up on Microchannel and offered EISA-based servers.
The more recent fight arose in the early 1990s as vendors demanded faster buses to replace the still-standard ISA bus in desktops and servers. Vendors needed faster buses, particularly for graphics cards that were taking a severe performance hit because they were communicating at the bus' slow 8.3 MHz rate when talking to the CPU and main memory. Vendors also saw that software applications could be speeded up. Several new technologies appeared, in addition to a new popularization of the EISA bus. The VESA Local Bus was synchronous with ISA, with a few more interconnects added to a regular ISA slot. A system could have two such slots, each with a more direct path to memory and the CPU, which would make it run faster.
Unfortunately, VLB didn't really improve the situation for nongraphics cards. Its competitor was the Peripheral Connect Interface (PCI), which offered a separate 32-bit, 33-MHz bus that ran faster and could carry more data. It was designed for any type of card and included the same features as EISA and more. In addition, it was more compact and designed to run at incrementally faster speeds.
Today, we see PCI as the only buses on most servers and the primary bus on desktops. A speed-doubled, quadrupled, or hexupled version is available in the Accelerated Graphics Port (AGP) slot intended for graphics cards. Even the fastest PCI slots now running at 66 MHz with 64-bit buses aren't enough for some applications.
Why did PCI win when Microsoft failed a generation ago with superior architecture? First, PCI was developed by a group of vendors, and is still operated under a joint agreement between many companies. Second, we had outlived the ISA bus' speed limitations due to faster processors and memory. Perhaps if we follow this same equilibrium of needs and wants, a new technology such as InfiniBand can survive.
Have we outgrown PCI?
It's time we admitted we've outlived the use of a shared bus architecture in which cards have to fight for bandwidth. Today's cards need too much bandwidth to accomplish their goals, but that can be fixed by making the bus faster or making several separate buses.
To run faster, we can either introduce more bits or run at a higher frequency. Adding more bits to the existing parallel card interfaces creates an even longer interface, which isn't practical from the user's and vendor's points of view. A better way would be to switch to a serial line as we do with modems, deconstruct the bits before sending, and reconstruct them after receiving. With a serial line, sending bits even faster is also possible. That is how InfiniBand works.
Running at a higher speed means computers will chew even more power, increasing the cost of the vendor's parts and lowering profits. With higher frequencies, the semiconductors have to be more accurate and, as the industry shows, that means lower yields. The parallel PCI interface and previous buses makes that a complicated problem because each bit line can cause interference in its neighbors. With higher frequencies, interference is greater.
More PCI buses are often added to PC servers, creating dual and even triple peer buses within the same server case. Although that works, limits are reached for space constrictions and efficient use of each bus. The best bet would be to have one card per bus and all the buses communicating with each other.
What kind of products will we see?
InfiniBand is designed for high bandwidth, so the first likely targets would be midrange to high-end PC servers, those with 4 to 16 processors and more than 4 GB of memory. Some of those are still rare but, as technology progresses, they become more accessible and usable by the masses. The first thing you'll notice is that the switched architecture can take the peripherals out of the computer's main box. Some products already follow that trend. For example, the Dell PowerEdge 6000 series has mostly room for the processors and main memory, requiring peripherals and storage to go into secondary units.
While breaking out the peripheral slots and storage into secondary units can introduce more potential failure points, the main processor node itself can be less complex. We don't need to support dual or triple peer PCI buses and hot-swappable disk systems in the main box. A smaller chance exists that maintenance on the drives or peripheral cards will cause trouble to the processor node's other internals. Without the drive system and any other moving parts within, the processor node needs fewer fans to keep it cool, consuming less power.
InfiniBand's switched architecture calls for several separate elements similar to that of Fibre Channel: the Host Channel Adapter, the switch, and the target bus units. The Host Channel Adapter (HCA) must be integrated into the system's motherboard and have direct access to the CPU and memory. What we'll likely see at the outset are HCA cards for PCI to allow you to upgrade without replacing your servers. That PCI card method doesn't take full advantage of InfiniBand because it can hit a bottleneck on the PCI bus. Thus, the HCA needs to be part of the server's central chipset, and the PCI bus must go through it to get to the CPU, memory, and disks, rather than the other way around. Also, instead of the SCSI chains connecting into the PCI bus either through a card or through an integrated chipset, it should now go over the InfiniBand system.
The HCA is just an end point. To have those multiple devices and buses connected to the InfiniBand system, you need a switch built into the server's motherboard. That internal switch multiplexes between the internal SCSI chains (if any), the PCI bus within the box (if any), and the rest of the computer. External devices could connect to that switch if there were separate, external ports for them, or they could connect through an external switch that links to the main computer.
You'll notice that the above description of buses inside the server is a first step with a server model that appears similar to today's servers, with CPU, memory, storage, and peripherals all in one box. The next step would entail moving peripherals and storage to separate units. In such cases, an internal InfiniBand switch wouldn't be necessary, and the processor node's HCA would have to connect to all those devices through an external switch.
What's next?
The midterm future, starting in mid-2002, will bring down InfiniBand component costs into the realm of smaller servers and workstations. The network capability of InfiniBand devices, using IPv6, means they can be accessed over a WAN and by many computers at the same time. That is a move away from local storage and peripherals to a configuration that's networked at the hardware level. We previously had to attempt that at the software level, using protocols such as NFS, CIFS, and Telnet.
InfiniBand may cause PC servers to catch up with minicomputers such as the AS/400 and even mainframes. The I/O channels in those other computers have given them a significant performance advantage in terms of guaranteed access to data. While the PC has been focused on raw computing power to handle I/O, minis and mainframes rely on controlled channels of data paths with guaranteed delivery within given time constraints. If the PC started to work that way as well, we might see a blur in the distinction across these computers, at least in terms of hardware. In software, the PC operating system will still probably take some time to catch up to those other systems' capabilities.
The biggest unknowns are the management models. By turning your server's internal network of peripherals and buses into an external one that can be accessed by multiple computers, you complicate the details of controlling those systems and providing proper access to users. The management software for today's networks may also become the general management tool for server hardware. Some, such as Tivoli and OpenView, are already well on their way to accomplishing that, although the complexities of InfiniBand may lead to a marked change in their feature sets as well.
Now that InfiniBand has been released, we'll see the crux of a new evolution of PC servers and peripheral devices. Who knows? In 10 years we may see no distinction between PC servers, minis, and mainframes. The differentiating factor will be support services and human-related factors. While PC systems were never intended to compete with those larger computers one-on-one, the PC revolution that began decades ago has brought down computing costs. With InfiniBand, we just might see it take over the whole industry.
About the author
Rawn Shah is an independent consultant in Tucson, Ariz. His latest book Unix and Windows 2000 Integration Toolkit (John-Wiley, 2000) is now available from Amazon, FatBrain and at major bookstores. Join him for his weekly online forum on UNIX-NT Integration. |
