InfiniBand Architecture (IBA) - Cliff Notes Over View
Here's an outright theft of some of the tutorial material, presented to build a framework for understanding and discussion. I hope it's not too technical, some granularity was sacrificed in the interest of condensing basic building blocks.
+++ Some Definitions:
IBA
A well-defined, layered architecture that provides:
- High Bandwidth / Low-latency communication with minimal CPU utilization
- Scalability
- I/O and Inter-process Communication (IPC) co-exist on the same fabric instance
- Transparent I/O adapter / device sharing
- Robust reliability / Availability / Serviceability
- Low-cost Hardware, excellent Price/Performance
Layered Architecture
IBA provides:
- forward and backward compatibility / interoperability
- allows layers to evolve at the rate technology evolves
IBA Link Defintions
MESSAGE
Applications, drivers, devices, adapters, etc. execute transactions via logical units of work termed messages.
Messages are moved via the IBA transport layer.
Hardware-based memory / resource protection to prevent unauthorized access.
Message semantics supported:
- Memory RDMA (Remote Direct Memory Access) Read / Write
- Channel Send / Receive
- Atomics (optional functionality)
- Multicast (optional functionality)
PACKET
IBA Hardware provides automatic message segmentation and re-assembly via packets.
End-to-end fabric unit of transfer.
Routable unit of transfer.
VIRTUAL LANE (VL)
Multiplex independent data streams onto a single physical link which provides:
- Differentiated services on a packet-boundary basis
- VL based routing across multiple paths
- Support for up to 16 VLs, with 1 VL reserved for fabric management services
- Implementation shall support a minimum of 1 VL application usage and 1 VL for fabric management
---- IBA defines a VL mapping algorithm to insure interoperability between end nodes which support different numbers of VLs
HOST CHANNEL ADAPTER (HCA)
Generally, a host controller that provides:
- Link Protocol Engine
---- Implements link protocol in hardware
---- Protocol engine present in each IBA component
- RDMA Engine
- Work Queue Engine
- Multi-port Load-balancing, Alternative Path (optional functionality)
- Hardware Transport Implementation
- Hardware Memory Registration and Protection
TARGET CHANNEL ADAPTER (TCA)
Attaches I/O endnode to the IBA Fabric, only implements what is required to minimally support the IBA and device-specific functionality.
SWITCH
Packets are switched within a single subnet using subnet unique Local Identifiers (LIDs)
Supports 1 to 15 VLs with configurable VL arbitration to provide QoS and limited Head of Line (HOL) blocking protection
ROUTER
Superset of IBA Switch functionality
Provides routing between subnets
Routers may join IBA to IBA fabrics
+++ Stacking the blocks of the IB Architecture, its fundamental structure looks like this:
[CPU1] - - - - [CPU2] - - - - [CPU3] - - - - [CPUn]
- - - - - [MEMORY CONTROLLER] - - - - -
- - - - - - - - - - - - - [HCA] - - - - - - - - - - - - -
- - - - - - - - - - - [SWITCH] - - - - - - - - - - - -
[TCA - FC] - - - [TCA - SCSI] - - - [TCA - ETHERNET]
Much of the spec work is nearing completion. HCAs and TCAs will be developed to spec by vendors. The Switch spec is the key element that springboards the IB market, and the IB Switch has got to be done right, it's got to be done quickly, and it's got to be fast, reliable, cheap, and
plentiful.
Looks like L'il Ole Ancor is in the middle of the field carrying the ball.
Douglas |
