Adding Multithreading to a Java Web Server

The last spec from cob spec that I got to pass is named TimeToComplete and is located within a separate sub-suite of tests called SimultaneousTestSuite.

It sends thousands of get requests to the server being tested and verifies that every response is a 200. It’s designed to break a single-threaded server (there’s even a note that says this), so the solution is obviously to make the server multi-threaded.

Multithreading allows you to split part of the execution of an application into smaller parts that can be run on separate threads at the same time*.

It may or may not actually be at the same time, depending on the number of processors available. If all the threads are within a single process, they will still appear to execute in parallel due to a feature called time slicing, where the process switches between threads very quickly.

Below are some of the things I learned about multithreading in general and about concurrency in Java, and how I incorporated that information into my Java server.

Process vs. Thread

A process is a type of self-contained execution environment that has its own runtime resources, its own memory space, etc. It can communicate with other processes, on the same system or another one, using IPC (Inter Process Communication) resources like pipes and sockets.

A thread is a kind of a light-weight process. Threads run inside a process and you can have multiple threads in the same process. A thread is also an execution environment, but it requires fewer resources than a process. One downside is that threads in the same process share the same resources, like memory space. Because of this, threads can’t perform any action that would interfere with those of another thread and/or need to have their actions synchronized.

Manually Creating Threads

You can explicitly create a thread in Java by creating an instance of the class Thread.

Thread thread = new Thread();

After creating a thread, you need to tell that thread what to do (i.e. what code it should execute) by providing the thread with a Runnable object* (an object that implements the Runnable interface).

Note — this is not the only way to provide instructions to a thread.

Thread execution can be paused by calling Thread.sleep(int timeToSleep) in the code running on that thread. This makes the processor available to another thread that might be running.

Although you can specify the time you want the thread to sleep, there are no guarantees that it will sleep for that exact amount of time (this is/can be limited by what the OS is capable of doing). Threads can also be interrupted, in which case an InterruptedException with be thrown.

Although creating threads is relatively simple, managing them is not. There are plenty of low-level tools for doing it yourself (synchronized, volatile, wait(), etc.), but it’s hard to do properly.

To make our lives easier, Java provides a concurrency utilities package to provide some higher-level abstractions with which to build concurrent applications.

Concurrency + Java Server

The concurrency abstraction that I used for my Java server is an interface called ExecutorService (specifically a ThreadPoolExecutor), which among other things provides a way to create a thread pool to which new tasks can be submitted. The ExecutorService is responsible for initially creating new worker threads, accepting tasks and either passing them to a worker thread, or if none are available, placing the task into a queue to be handled later.

It also provides some other handy features like the ability to stop accepting new tasks but still complete all that have already been submitted (or to shut down immediately), as well as the option of returning a Future, which can be used to check the status of some asynchronous task.

For my situation, I was mostly concerned with its thread and task management capabilities.

Even though I was aware of the general direction I needed to go, figuring out how to incorporate multiple threads into my existing server required re-evaluating what the server’s responsibilities ought to be.

Specifically, I needed to separate the process of accepting the incoming connection and creating a client socket from what I do with that connection (read the request, send it to the application and ask for a response, send that response back through the connection and finally close it)

This was originally all happening in a single method. I split it out into several smaller methods to make it easier to test, but adding multiple threads required the creation of a new class, called ConnectionHandler, both to pull that separate responsibility out of the server but also because it needed to implement the Runnable interface to be passed to the Executor’s execute method.

Here’s a snippet of the new code:

ExecutorService executor = Executors.newFixedThreadPool(100);
ServerSocket server = new ServerSocket(myPort);

The code above happens before the loop below

while (true) {
    Socket socket = server.accept();
    Runnable connectionHandler = new ConnectionHandler(socket, app);
    executor.execute(connectionHandler);
}

The first line (inside the loop) is where we accept the request from the client.

The second is where we create a new ConnectionHandler, giving it the newly created client socket (this is where we’ll get our input stream) and the application, which contains the logic for figuring out what to send back to the client.

On the last line we call the ExecutorService’s execute method and pass it a Runnable object. The Runnable is basically a wrapper around the code we want to execute on a new thread, placed inside a method a method called run.

The code above runs in the program’s main thread. It’s important to keep that thread free and unblocked so that it can continue to accept incoming requests. We want to avoid refusing connections, if possible.

Even if the program gets more requests than it can respond to in a short amount of time, the requests are still accepted and put into the ExecutorService’s queue to be handled as soon as a worker thread is available. The client may have to wait a bit longer for a response, but a response will eventually get there.