Try this article for some background reading J. It is more technical than most and might help answer questions about signal loss and why analog engineers are required. The advantage in IB development would go to those companies already well stoked.
glenn
eoenabled.com
Advanced I/O still comes up short
By Deepal Mehta, Business Unit Manager, Computer Servers,
William Lau, Senior Manager, CoreWare Development &
Applications, Internet Computing Division, LSI Logic, Milpitas, Calif.
EE Times
(05/22/00, 1:00 p.m. EST)
The Internet and burgeoning e-commerce applications
are placing major demands on server I/O performance.
The PCI and PCI-X buses are expected to meet today's
I/O requirements. Infiniband, which is next-generation
I/O technology, promises higher bandwidth and
scalability to meet emerging I/O needs. It also offers
better reliability and quality-of-service (QoS) capabilities.
Yet even this advanced I/O technology comes with its
own set of system design issues and challenges.
Against this backdrop, there's also the problem that I/O
performance isn't keeping pace with CPU performance.
Server OEMs are focusing on I/O and memory
subsystem designs to differentiate their server platforms.
Consider that designers still face signal-integrity and timing issues designing
PCI I/O at the 66-MHz level. The PCI-X specification permits the I/O design
to move to a maximum of 133-MHz, 64-bit performance. That is double the
bandwidth of the 66-MHz, 64-bit PCI bus, resulting in 1-Gbyte/second
performance.
Still, issues involving the areas of timing performance and signal integrity
continue to arise. Here, the I/O interface is pushed to the limit. For example,
the clock to Q output propagation delay is 6 nanoseconds. The PCI-X spec
calls for a 3.8-ns path delay with basically the same logic element. Setup time
is 3 ns for the PCI 66 and only 1.2 ns for PCI-X. Also, there is a 7.5-ns cycle
time between input and output registers for the core logic to propagate. That
core can easily have 15 to 20 levels of logic. So core logic timing is more
relaxed due to the PCI-X protocol. But the I/O is much tighter.
The PCI-X 3.8-ns clock to Q output delay means the PCI clock comes into
the ASIC device through the input receiver, to a phase-locked loop (PLL),
clock driver, on to the clock input of the output latch. That output flip-flop
then goes through a JTAG mux and finally through a PCI-X driver. This path
not only includes the delay of each element, but it also includes PLL jitter,
PLL offset and clock skew. All these must be achieved within 3.8 ns to meet
the PCI-X spec.
Critical timing
Setup time is the other critical timing area. Data enters the ASIC, goes through
the input receiver, through an input JTAG mux and on to the input register for
clock arrival setup. These data transactions must be achieved in only 1.2 ns,
and must also account for PLL jitter and clock skew.
Perhaps the most critical issue is the design of the PCI-X I/O circuit's
input/output receiver path. The driver definitely cannot be too slow.
Otherwise, the designer cannot meet timing closure. But the tricky part is that
it cannot be too fast either, because in a high-speed system the designer must
pay close attention to signal integrity and noise problems. The PCI-X bus by
design is a noisy bus since the signaling is based on the reflection of the
transmission line wave. Plus, it is a long unterminated bus.
When designing the driver, the design engineer must consider the signal
integrity and overall timing budget of the system, as well as the ASIC. For
instance, let's consider the ASIC design and the driver part of the delay path.
If the driver operations consume 3 ns, only 0.8 ns remain for the receiver. So
the burning issue is how do you budget the delay for the driver and for the
receiver? The design engineer has to consider that issue carefully.
On the other hand, the designer can make the buffer fast by reducing the
propagation delay. However, lower propagation delay typically results in a
higher slew rate, which makes the buffer noisy. The designer needs a balance
between propagation delay and slew rate. Signal-integrity issues, therefore,
must be taken into consideration to make sure the buffer falls within the PCI
and PCI-X specs and not be too noisy. At times, a buffer design can fully
meet the PCI-X spec, but can still be very noisy. It can still have high current
due to the pre-driver stages. In that regard, it is important for the designer to
have sufficient previous PCI design experience to execute a well-balanced
circuit.
Infiniband uses point-to-point serial interfaces. That approach provides
several benefits at the physical layer over parallel multidrop technologies that
have been used in I/O buses traditionally (e.g., PCI and PCI-X):
In a point-to-point signaling environment there is only one driver (at one end
of the wire) and one receiver (at the other end of the wire). This results in a
clean transmission path where signals may propagate down the wire at high
speeds with minimal distortion and degradation. Multidrop buses, on the other
hand, have multiple drivers and receivers hanging off the same wire at different
points along the wire. Each driver and receiver introduces transmission-path
imperfections (such as parasitic inductance and capacitance). Those
imperfections result in signal reflections, distortion and attenuation.
Consequently, the maximum speed at which you can drive a signal in a
multidrop environment is lower than in a point-to-point environment.
Infiniband employs serial connections with embedded clocks and lane
de-skew (for x4 and x12 Infiniband) on the receive side. With this
architecture, you don't need to worry about clock-to-data skew because the
clock is actually embedded in the data. With lane de-skew you don't need to
worry about skew between data lines (as long as you keep that skew within
what is specified in the Infiniband standard) because any skew that has been
introduced is eliminated on the receive side through realignment techniques.
These two features let you go longer distances because you don't have to
carefully manage the skew between clock and data, and between data lines.
Server designs
From another front, the designer must factor in the effects next-generation
process technology will have on server I/O designs. PCI and PCI-X are
parallel buses that require a number of I/O pins. The PCI-X device requires
64 data and address pins and another 16 command pins for 1-Gbyte/second
bandwidth. As semiconductor technology moves to 0.18 micron and below,
the number of pins required by PCI-X may make the design pad limited. This
limits the die-size reduction achievable through next-generation process
technology. The Infiniband standard, on the other hand, is a serial technology
and thus offers considerably more bandwidth with a lower number of pins.
Essentially, Infiniband is a switch fabric-based architecture. It decouples the
I/O subsystem from memory by using channel-based point-to-point
connections rather than a shared bus, load and store configuration. The
interconnect uses a 2.5-Gbit/s wire-speed connection with 1, 4 or 12 wire link
widths. One wire provides theoretical bandwidth of 2.5 Gbits/s, four wires
provide 10 Gbits/s and 12 wires provide 30 Gbits/s bandwidth. Hence,
Infiniband provides system designers scalable performance through multiline
connections, as well as a host of interoperable link bandwidths.
In an Infiniband server node, there is an Infiniband I/O bridge rather than the
PCI I/O bridge. Also known as an Infiniband host-channel adapter, the
Infiniband bridge generates Infiniband links, which are then connected to the
Infiniband network, which may consist of an inter-processor communication
(IPC) network, or storage area network (SAN) or local-area/wide-area
network (LAN/WAN) (Fig. 1). Various I/O subsystems like Ethernet, Fibre
Channel, SCSI or even interprocessor communication communicate through
the Infiniband network switch fabric.
Migrating a PCI-X server I/O design to Infiniband involves several
considerations. A key decision is whether to embed the serial transceiver in
the ASIC or keep it external as a separate PHY. Of prime importance is a
power analysis of the device since an embedded transceiver adds power
consumption. On the other hand, embedding the serial transceiver in the ASIC
requires fewer pins, which may lead to smaller die-size and package option
and hence lower cost. So here the designer is dealing with cost/power
trade-offs.
The designer also has to evaluate power consumption in terms of packaging
and carefully determine the type of package best suited to the transceiver,
whether it be a standalone or embedded device.
Another consideration is the number of wire implementations the design
requires. Is it 1, 4 or 12? A big part of this consideration comes from
matching bandwidth with memory bandwidth on the server node, as well as
with the performance the designer is attempting to achieve.
Interoperability is key to any new technology. Infiniband is no exception and
poses certain challenges to the designer to make sure the server I/O Infiniband
subsystem interoperates with other subsystems in the enterprise.
End-to-end interoperability from host-channel adapter to storage or
networking element through the Infiniband fabric is very important. It is also
important that the selected operating system support Infiniband.
Infiniband is still in its infancy as a new technology, and the spec is still
evolving. So, rather than hardwiring a device, the designer may choose to
have an embedded microcontroller within the chip to gain extra programmable
intelligence.
Spec evolves
Otherwise, with a state machine implementation, the system designer must
keep changing it as the spec evolves. But with the programmability a MIPS or
ARM microC core provides, a designer can be more flexible with
implementing the Infiniband protocol. Embedded intelligence in the target
devices also provides the traditional function of minimizing the interrupts,
reducing bus traffic and offering a better error-recovery mechanism.
Signal integrity is another area the designer must evaluate. Since Infiniband
touts a 2.5-Gbit/s wire speed, signal integrity is paramount.
Serializer/deserializer (Serdes) that operate at these speeds often use
expensive bipolar, GaAs or BiCMOS process technologies. Few chipmakers
offer Serdes using standard CMOS process technology, and even fewer can
embed this Infiniband technology in a large ASIC and do it cleanly with good
signal integrity.
In addition, having a robust receiver that can receive signals without errors that
have been attenuated or distorted is important.
Some design camps want to completely switch to Infiniband. Other OEMs
don't believe PCI-X will go away soon and believe that PCI-X and Infiniband
will co-exist until the end of this decade. Hence, there will be system engineers
who will opt for a hybrid system that supports both PCI and Infiniband. This
calls for a system architecture that supports PCI-X for legacy functionality, as
well as Infiniband.
See related chart img.cmpnet.com |
