Networks

Fast interprocessor networks are together with fast processors the decisive factors for both good integrated parallel systems and clusters. In the early days of clusters the interprocessor communication, and hence the scalability of applications, was hampered by the the high latency and the lack of bandwidth of the network (mostly Ethernet) that was used. This situation has changed very much and to give a balanced view of the possibilities opened by the improved networks a discussion of some of these networks is in order. The more so as some of these networks are, or have been employed also in "integrated" parallel systems.
Of course Gigabit Ethernet is now amply available and with a maximum theoretical bandwidth of 125 MB/s would be able to fulfill a useful role for some applications that are not latency-bound in any way. But with latencies in the order of somewhat less than 100 µs and in-switch latencies of 35--45 µs the applicability is too restricted to be the network of choice (except perhaps for price reasons). We restrict ourselves here to networks that are independently marketed as the proprietary networks for systems like those of IBM and SGI are discussed together with the systems in which they are incorporated. We do not pretend to be complete because in this new field players enter and leave the scene at a high rate. Rather we present main developments which one is likely to meet when one scans the high-performance computing arena.

A complication with the fast networks offered for clusters is the connection with the nodes. Where in integrated parallel machines the access to the nodes is customised and can be made such that the bandwidth of the network matches the internal bandwidth in a node, in clusters one has to make do with the PCI bus connection that comes with the PC-based node. The type of PCI bus which ranges from 32-bit wide at 33 MHz to 64-bit wide at 66 MHz determines how fast the data from the network can be shipped in and out the node and therefore the maximum bandwidth that can be attained in internode communication. In practice the available bandwidths are in the range 110--480 MB/s. Since 1999 PCI-X is available, initially at 1 GB/s, in PCI-X 2.0 also at 2 and 4 GB/s. Coupling with PCI-X is now common in PC nodes that are meant to be part of a cluster. Recently PCI Express became available. This provides a 200 MB/s bandwidth per data lane where 1×, 2×, 4×, 8×, 12×, 16×, and 32× multiple data lanes are supported: this makes it amply sufficient for the host bus adapters of any communication network vendor. So, for the networks discussed below often different bandwidths are quoted, depending on the PCI bus type and the supporting chip set which bandwidth could be attained. Therefore, when speeds are quoted it is always with the proviso that the PCI bus of the host node is sufficiently wide/fast.