Everyone’s trying to exploit meltdown, and …
here we are just taking advantage of existing flaws in timing to find internet scale bugs that nobody cares about.
In security, we’re often confronted with challenges and sometimes those challenges may seem impractical to tackle at the time. Timing attacks are one of those things that are at the very heart of the recent wave of branded exploits and while they take all the spotlight we’ve got a long way to go to when it comes to tackling the scope of this class of vulnerability.

Timing has been used to glean information since the beginning of signals intelligence but the recent rash of cpu vulnerabilities have savvy technologists scratching their heads. Lucky for us this isn’t about those attacks, or rather this is not about those specific attacks it’s about the entire class of vulnerability known as: “Timing Attacks”
I’ve been fascinated by timing side channels since I first began understanding them. The concept is simple enough, computers sometimes perform operations and want to keep some component of the process a secret. Due to differences in how long certain things take, it is often possible to rediscover this information simply by profiling the time of two separate requests.
First we start with my personal favorite, the attempt to prevent user enumeration that appeared after people realized letting an attacker know if an account exists is probably a bad idea. This goes back to the dawn of multi-user systems, and login pages in general.
If, for instance you have a login process that discloses:
> user fuu pass bar
username incorrect.
> user foo pass bar
password incorrect.It becomes trivial to request a series of logins for a long list of common usernames, and upon finding a password incorrect field, you now know that you have found a valid user.
The fix for the above, in a great many systems went something like the following pseudocode:
function check_login(username,password):
if (!checkuser(username)):
return False
elif (checkpass(username,password)):
return True
else:
return Falseif (!check_login(username,password)):
print "unknown username or password"
This concept, and conditional likely exists in a great many authentication systems, not just because of the historical context — but because it is one of the easier ways to implement this. What this means for us is software that follows this design pattern will expose this vulnerability.
Apache Httpd and Basic Auth
Apache was the staple web server for many years, and still represents a sizable portion of hosted content on the internet. Apache2 has a simple authentication mechanism (basic auth) that we will use to demonstrate this vulnerability. The premise is, the system wants to return the same result (username or password not found is a great example) to an attacker whether a user exists or doesn’t exist.
And by setting the result HTTP/1.1 401 Unauthorized and something like the below, that’s sufficiently vague you mask the result:
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn’t understand how to supply
the credentials required.</p>
Unfortunately as we surmised above: the order, and logic of conditionals in the authentication process provides some information that we were trying to hide. Enough, it turns out, to be able to predict which accounts exist on the system and which don’t. Now why do I say predict? Well, unfortunately even in this very simple case, there exists a margin for error. Still the information leakage is there and we’re going to go about finding it.
First we look at the code that defines check_password which gets executed as part of the credential validation process. As we walk through the function, there are a series of conditionals that return AUTH_USER_NOT_FOUND before they get to the code that checks the validity of a password. And while they are returning that value, the display message is not changing on the user’s side, but that doesn’t change the case: only real users actually run the password verification function.
while (!(ap_cfg_getline(l, MAX_STRING_LEN, f))) {
const char *rpw, *w;
/* Skip # or blank lines. */
if ((l[0] == '#') || (!l[0])) {
continue;
}
rpw = l;
w = ap_getword(r->pool, &rpw, ':');
if (!strcmp(user, w)) {
file_password = ap_getword(r->pool, &rpw, ':');
break;
}
}
ap_cfg_closefile(f);
if (!file_password) {
return AUTH_USER_NOT_FOUND;
}
AUTHN_CACHE_STORE(r, user, NULL, file_password);
status = ap_password_validate(r, user, password, file_password);
if (status != APR_SUCCESS) {
return AUTH_DENIED;
}The above is an exerpt from: thttps://github.com/apache/httpd/blob/modules/aaa/mod_auth_basic.c
Take a look at the bolded sections, the first in the execution is a return when a username fails to be found in the htpasswd file. The second is the password validation function, that only gets run if an authentication attempt makes it through the guantlet above. These two chunks of code will help us leak a single bit of data at a time.
Password checking functions are typically going to take much more time than the user verification step, so this will probably give us two different timing cases to analyze. Based on this assumption we verify with simple spot checks and they show a roughly 20% difference in a valid vs invalid user.
Now that we can see the function is vulnerable to timing analysis, we need to figure out whether exploitation is practical. In order to determine the degree this was susceptible we need to get the first piece of the puzzle. We need a timing oracle. A timing oracle in this case is a request we can count on to return a referential timing result.
Since it’s not reasonable to expect a certain account to exist on every httpd server we test, we can instead create a random string and *hope with confidence* that there is no user likely to match a random string (something like CELASKKMR2ASDDAB2312.) Now we try logging in a hundred times with that account and record the timing. After we throw out the outliers and take the average of the requests, the average timing of the request becomes our guess for an account that doesn’t exist.
Now the smart thing to do here, would be to calculate exactly how long each individual step of the request takes and get really detailed timing profiling. But we’re here doing `gud enuf for sploit` work… Instead of doing the smart thing, we instead applied the same test as above with a sample set of known and unknown users.

Taking the difference between the two timing runs (the oracle and the tested account) we just set an arbitrary ceiling and turn the nob until we see all of the known accounts. If we wanted to perform internet wide scanning, we would need to be much more methodical, but since this is simply for lab prototyping: setting an upper and lower bound and sliding them back and forth has worked to accurately predict accounts.
So, to be certain we’re able to perform these attacks using this blunt approach we’ll try them on some very low latency links first where timing variance will make up a much larger percentage of the actual round trip time.

That also means in the wild we need to gather enough statistics to get over the jitter of whatever connection we are testing on. For the purposes of this demo I used a local loopback server, and tested on amazon network nodes adjacent to each other (2.5~ms), which minimizes the network noise issue. I’ll leave the noise floor solution and dealing with latency as an exercise for later.
Now, we have a methodology, and a design laid out. Using this information we were able to implement a prototype exploit using command line binaries and python profiling. It was very cloodgy and required using some double escaping that would make anyone’s eyes bleed. But it was effective and managed to work on the test platform.
It’s alive!
And with that test we now we have a tool that can reliably detect this vulnerability through observation. Once I had confirmed the existence of this vulnerability I drafted up a quick email and fired it off to the apache security team.
(If you’ve never tried to find the right folks to report vulnerabilities to, you may never have read the saddest google search ever: https://www.google.com/search?q=how+do+I+report+a+vulnerability. I had plenty of good advice on what to do here from lots of veteran bug hunters, and I still felt like I was sending my message into the void.
A couple weeks or so later the Apache security folks kindly informed me they didn’t think it was really worth fixing.
based on this being basic auth, the highly suspect mechanism to transmit
users and passwords … the project would conclude vectors against such a suspect login mechanism do not constitute a vulnerability.
And to be quite honest, I can’t really disagree with this statement too much, the feature is sort of “best effort” in terms of security. Though I think it’s at least worth the effort of attempting to document a fix. Later it’s suggested there should be an update to the guidance documentation indicating this is a well known weakness in this sort of auth system.
This seems logical, as fail2ban or an equivalent could solve this problem by blocking multiple failed login attempts. Unfortunately this requires a third party tool, and isn’t something you can easily toggle as a setting in Apache httpd.
Struggling with this dilema, and the prospect of more work. I will finish this chapter of the writeup with this piece left hanging: For a clever person, usernames in basic auth are innumerable, but they are also enumerable.