Introduction
Over years of computer evolution, there has been a progression of systems from single core to multi-core processors enabling the concept of parallel processing. Parallel processing enables computer systems to truly execute multiple tasks at the same time due to the presence of multiple processors/cores.
With this ability, comes the concept of parallel programming where developers write programs that break down larger problems into smaller, independent, often similar parts that can be executed simultaneously by multiple processors communicating via shared memory, the results of which are combined upon completion as part of an overall algorithm.
Supercomputers and high-performance computing clusters aren’t the only devices that can use parallel programming. It’s also often used in regular technologies.
For example, contemporary web browsers use parallel processing to handle network requests, render HTML/CSS, and execute JavaScript concurrently across several CPU cores, resulting in quicker web page rendering.
Video game engines use parallelism to mimic real-time AI interactions, physics, and complex landscapes.
Parallelism vs Concurrency
There is often a confusion between parallelism and concurrency as they appear on the surface to mean the same but in the computing world, the two terms have slightly different meanings.
Concurrency conveys an illusion of parallelism during processing, in that one core is in action but allows for overlapping and interleaving of multiple tasks/threads in an unnoticeable short amount of time as though they were being executed at the same instant whiles parallelism involves multiple processors executing different tasks at the very same time.
We can picture concurrency and parallelism as a cooking session. Say you want to prepare a buffet for a group of people, so you hire one chef (representing one processor) to make all the meals.
Ideally, our chef can only do one thing at a time, he cuts up some vegetables at one point, boils something at another and fries something at another point (representing tasks).
The chef by some magic is doing all three activities at the same time but in truth, he can only handle one thing at one point in time. Contrary to one chef preparing all meals, you could hire multiple chefs for the preparation and have different actions undertaken by different chefs at the same time (true parallelism) which would produce a faster output under ideal conditions.
That is how processors work. In truth, a processor can only execute one task at a time, but the processing time is so quick that when concurrency is being employed, it seems as though multiple tasks are being executed in that instant.
We can achieve parallelism with multiple processors where one task is divided into multiple threads and executed across different processors and the result presented as one after execution to further reduce processing time and increase efficiency.
Programming In Parallel
Java provides various mechanisms to enable developers write parallel programs efficiently.
1. The Fork/Join Framework: Java’s Fork/Join framework introduced in java 7, enhances program performance by using all the available processor cores of the system and follows a divide and conquer approach to process programs in parallel.
2. Custom Thread Pools: With the “ExecutorService” interface, developers can create custom thread pools and specify the desired level of parallelism based on the CPU cores available
3. Parallel Streams API: Java enables developers to parallelize operations on collections by leveraging multicore CPUs by dividing work among threads. The parallelization is automatically handled, but the order of the operation results is not guaranteed.
Java program using parallel Streams:
In the code above, we are generating an array of random integers and finding the maximum value using java’s stream API. The parallel implementation is taken care of for us and we see that the same results are yielded for both stream types.
You notice that the time taken to complete each task changes as often as you run the code, sometimes with the sequential stream taking a shorter time. Significant time difference is seen between parallel and sequential streams in applications that use a large complex dataset and perform more complex operations.
Advantages of parallel programming
1. Enhanced speed and efficiency: Complex calculations and data-intensive procedures are performed more quickly when activities are divided over several processing units than when they are programmed sequentially, which cuts down on computing time.
2. Resource Utilization: With multiple processing units, there is high use of resources in multi-processor systems ensuring that no processing power goes unused, and the full potential of hardware resources is harnessed.
3. Scalability: Parallel programs can scale efficiently to handle larger datasets or more complex computations. As the number of processing units increases, parallel programs can distribute tasks across these units, effectively utilizing available resources and maintaining performance as workload size grows. This scalability is essential for applications dealing with big data or high-performance computing.
Disadvantages of parallel programming
1. More resource requirements: Contrary to sequential programming that requires only one processing unit to execute a task fully, parallel programming entails the utilization of at least two processing units to achieve parallelism which may or may not be necessary depending on the use case. Parallel programs often require more memory to store intermediate results, communication buffers, and synchronization primitives. Each parallel task may need its own data structures and memory space, leading to increased memory consumption compared to sequential programs.
2. Communication overhead: In parallel programming, tasks often need to communicate with each other to share data or synchronize their execution. This communication overhead can be significant, especially in distributed parallel computing environments where tasks may be running on separate machines connected via a network. Communication involves data transfer, serialization, deserialization, and synchronization mechanisms, all of which require additional computational resources.
3. Complexity of designing, coding and debugging: It requires a great deal of understanding to efficiently write programs that will execute in parallel as compared to sequential programs. With the non-deterministic behavior of parallel programming due to interleaved execution of parallel tasks, it is challenging to reproduce and diagnose bugs.
Key Considerations
It is important for you to consider whether parallelism is required for your challenge and whether the possible benefits outweigh the increased complexity if you wish to use parallel programming regularly.
Here are some crucial things to remember:
1. Determine Which Tasks Can Be Parallelized: Not All tasks are best performed in parallel. Decide which portions of your program can be parallelized successfully, then concentrate on making those areas as efficient as possible. Seek tasks with a high computational demand and few dependents.
2. Measure & Benchmark: Evaluate the sequential version’s performance prior to parallelizing the code. When deploying parallel solutions, benchmarking gives you a baseline for comparison and aids in identifying bottlenecks.
3. Begin with basics: Start with small-scale parallelism and add more complexity as you go along. To become comfortable with parallel constructs and prevent overwhelming complexity, experiment in a controlled environment with libraries.
4. Scalability of Mind: Take scalability into account right away. Make sure the problem’s size and the hardware resources available can be handled by your parallel solution efficiently. Seek solutions that scale well with varying numbers of processors and levels of workload.
Written by Emma Naa Kai Odametey.
