Many people mistake parallel computation for high performance computing (HPC). They are however different concepts.

  • Parallel computing means running multiple elements of a system simultaneously. In many situations, this is both necessary and sufficient.
  • HPC emphasizes having each execution element make optimal use of its environment. When parallel computing is no longer sufficient (e.g., the problem becomes too large or the required infrastructure becomes cost prohibitive), HPC tactics must be employed.

The Parallel Paradigm

With the advent of multi-core systems in the mid-2000s, parallel computing proliferated. Applications needing higher throughput or lower individual transaction latency simply could divide and conquer: divide the workload among multiple application instances and combine the multiple independent result streams, if necessary. As long as the problem lends itself to subdivided operation, this is quick and easy parallel computing, needing little more than scripting to effect. However, parallel computation does not scale well. Adding more instances to support a higher workload burdens the operating system (OS) and system resources with considerable overhead and redundancy.

One alternative approach is distributed computing: multiple instances on multiple machines. Coupled with parallel computing, this paradigm can squeeze out additional performance. However, distributed computation has its limits as well. Additional hardware requires additional money: this can become cost prohibitive, as one must factor in the capital expenditure, power costs, and maintenance. Moreover, we have now introduced additional overhead – coordination among the servers as well as input/output congestion.

The reality is that parallel computing says nothing about resource utilization efficiency. Tools such as Apache Spark facilitate implementing parallel/distributed solutions, thus lowering the barrier to entry, but fail to address the root issue – resources are not free. Each additional instance (e.g., process, thread, or server) imposes an overall system cost.

  • An additional instance requires memory and computing cycles to be siphoned away from all other instances and from all other applications attempting to run on the machine.
  • An additional machine requires capital to be siphoned away from all other productive activities.

There is no infinitely deep pool or resources from which to draw.

 The HPC Solution

High performance computing starts from the premise that resources are scarce, focusing on efficiency to achieve performance.

Removing OS intervention reduces system overhead. Because OS’s have an extremely large and varied set of responsibilities, they are decent at everything but not great an anything; removing their intervention can significantly improve performance. The following are among the strategies HPC uses to limit OS intervention:

  • Use long-lived threads, as opposed to multiple processes or short-lived “transactional” threads.
  • Pin threads to cores with real-time operation.

Computation speed is optimized through hardware-aware data structures and algorithms. HPC is thus able to exploit hardware capabilities such as single instruction / multiple data (SIMD) operations or the presence of accelerators (e.g., GPGPUs) or co-processors (e.g., Intel Xeon Phi). Moreover, all system and application code is both lock- and wait-free for maximum core / CPU utilization.

HPC minimizes memory utilization through the following mechanisms:

  • Data locality: Keep data handy only to the compute resource using it.
  • Cache management: Minimize cache misses and false sharing on each core cache. This is also necessary for enabling SIMD.
  • Shared memory: Exchange data among threads using shared memory. This also helps to minimize OS intervention and optimize intra-node / inter-thread communication.

Also, it should go without saying that there can be no memory leaks!

 Big Denominator

Big Denominator offers simple to use, off-the-shelf access to HPC tools. Clients gain access to all the above-mentioned features while staying focused on their particular business problems.

  • For developers looking to add HPC to their solutions, we offer EDGE, a fully featured development kit.
  • Our contract/consulting services offer customized solutions to meet specific client requirements.