DANGER ! This page is 3 or 4 years out of date and is no longer applicable for the state of the art in Linux. This page reviews the situation as it stood circa 1995 and 1996.

MP and Clustering for Linux and Related Info

High-performance computing can be broken into two market segments: Commercial and Scientific. This distinction is important, because the two segments have their own special requirements, needs and applications.

Commercial computing usually focuses on large databases, large application servers, high-performance transaction processing, accounting, process flow and management systems, database mining. Commercial computing tends to be integer intensive, disk and i/o intensive.

Scientific computing focuses on simulation of large, complex dynamical, physical systems, and on data visualization. Scientific applications tend to be floating-point intensive, and place a premium on high-bandwidth interprocess communications. Some industries, e.g. petroleum exploration, handle large quantities of data *and* need lots of CPU power.

The hardware and software needed for each application may seem superficially similar, but in detail becomes wildly divergent. That said, there is some overlap in general support. Things like SMP (Symmetric Multi-Processing or Shared Memory Multi Processing) and multi-threading fall into the General Computing category.

General Computing

The work that was done by the Linux SMP project to enable Intel MPS (version 1.1 and 1.4) compatible hardware is now a mainstream part of the 2.x kernel distributions. Some additional status can be found at Erich Boleyn's Status Page. The Parallel-Processing HOWTO pages provide a thorough introduction to the technical terms and concepts, as well as the status of Linux parallel processing (for both SMP and clusters). A variety of threads packages, many of which can be scheduled over SMP, are available. These and other basic questions about threads are answered in the Linux Threads FAQ.

High Speed Networking

• Compu-Shack offers a FDDI card. For those who can't read the German web page, click on the word "linux" to get the drivers.
• Myricom offers the Myrinet series of affordable giga-bit network gear. Of particular interest is the list of customer projects employing Myrinet technology.
• The ANSI/IEEE Scalable Coherent Interface (SCI) standard defines a means of creating closely linked clusters.
• There is a a beta version of a driver for the FORE Systems PCA-200E 155Mbps ATM adapter. Includes mailing list, detailed instructions.
• A FDDI driver for DEC FDDI cards is included in the version 2.0.24 and above kernels. Written by [email protected].
• An overview of Shared Memory Clusters can be found on the Dolphonics web page.
• Parastation offers a high-performance hardware interconnect card, intended for creating parallel clusters.
• The Multiple Disks HOWTO now includes a section on disk management in clusters.

Commercial Computing

Databases

Several high-availability, redundant-disk, SMP-enabled databases seem to be available for Linux. One is from Solid Technology, another from Software AG. See the Database Page for details. I have not been able to verify if the SMP and redundant-disk and clustering features are supported on Linux.

pWEB

The pWEB Parallel Web Server Harness will distribute URL requests to multiple servers based on load and/or URL, for load balancing or I/O balancing. The harness can be used with most web servers.

Inktomi

Inktomi is a massively-parallel web search engine.

AOL Server

The AOL Server for Linux web server from America OnLine will schedule multiple threads across multiple processors.

Scientific Computing

• The Linux High-Performance Computing page lists extensive resources for MP, clustered, distributed and parallel solutions for Linux.

• Purdue University's Prof. Hank Deitz maintains a Parallel Processing for Linux. This includes info on Linux clusters and SMP.

• The Beowulf project at Nasa attempts to scale Linux into the giga-bit per second range. The work seems to focus on bonding together fast Ethernet cards, and running distributed processing systems such as PVM (Parallel Virtual machine) or RPC's. They hope to develop NVM (networked virtual memory -- distributed shared memory -- distributed over the net), and MPI (Message Passing Interface Standard).

• There is a port of Linux to the 16-node SuperSparc-based AP1000+ Fujitsu Supercomputer. Can anyone point me to a similar project for the Intel processor? Any hope of this on the IBM SP?

• The Gamma project is attempting to develop a high-performance message passing interface on the raw hardware in order to create a fast cluster.

• The Legion Worldwide Virtual Computer is attempting to define the infrastructure necessary for a petaflop wide-area computer.

DANGER ! This page is 3 or 4 years out of date and is no longer applicable for the state of the art in Linux. This page reviews the situation as it stood circa 1995 and 1996.