A call for better application (security) logs

TL;DR poor quality of application (security) logs prevent early detection of attacks and make investigation challenging at best and inconclusive most often. With cloud provided decreased cost of storage and flexible computing capacity, there aren’t many excuses left to not do good logging.

Anyone who has looked at web server logs during a security incident has almost universally lamented the lack of details in them. The example log entry in the “Apache Common” format displays a few useful details, but omits even more, such as:

• Was the request _actually_ successful? All we know from the status code that there wasn’t an error.
• Which user issued the request?
• Was the request forwarded and so — what was the true IP address?

10.185.248.71 - - [09/Jan/2015:19:12:06 +0000] 808840 "GET /inventoryService/inventory/purchaseItem?userId=20253471&itemId=23434300 HTTP/1.1" 200 17 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

Some of it, such as including “X-Forwarded-For” header to show is easy to capture from the web server configuration. You could even try and parse out Cookie from headers such as “JSESSIONID” to try and identify at least to which session the request belonged. But how to deal with a question “did the request actually succeed and who was the user”? This information is available only to the application and only if it is constructed to log well.

This all becomes quite important when you are looking for attacks on the application. Consider this example entry:

10.185.248.71 - - [09/Jan/2015:19:12:06 +0000] 808840 "GET /inventoryService/inventory/purchaseItem?userId=' OR 1=1 '&itemId=23434300 HTTP/1.1" 200 17 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

A trained eye will quickly spot an attempt at SQL injection. Is it a cause for alert? Web is a noisy place and anyone who’s looked at an Internet exposed web server logs will quickly identify _tons_ of indiscriminate hacking attempts. But did the attack succeed? There is no easy way to tell from this line alone and the security analyst will have to try and see variation in response sizes for the same URL, preceding and following request lines (if an attacker is retrieving the table names, it’s a good indication attack worked). An art similar to reading tea leaves.

Taking it further, let’s consider another _very_ common pattern in modern applications — the API pattern, which would leave logs similar to this:

10.185.248.71 - - [09/Jan/2015:19:12:06 +0000] 808840 "POST /inventoryService/inventory/purchaseItem" HTTP/1.1" 200 17 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

As the method is POST and the parameters are in the body of the request, this isn’t typically logged, because the web server needs to decode a multipart message. While _it could_, it doesn’t mean that the application will act on it the same way which gave rise to an exciting category of web application attacks — the HTTP Request Smuggling.

All of this is exacerbated by the layered deployment of applications. In a typical modern deployment example, a user’s request could be proxied and routed via:

1. DDoS protection (e.g. Cloudflare, AWS WAF,)
2. A CDN (e.g. Cloudfront)
3. A front-end load balancer
4. A web application firewall
5. Front-end router/handler (e.g. a web server)
6. Back-end load balancer
7. Back-end microservice itself.

It isn’t necessarily clear where is the “best” place to capture what where and to be useful to security analyst (and the DevSecOps engineer perhaps), the logs have to be somehow coherent. For example, what if we tag a request with a unique ID early on and carry that throughout the delivery chain?

I’ve put together a mind-map of things that I think should be captured along with some concerns:

What should be captured by “web application logs”

Now, I am under no illusion that teams will rush to rewrite their stacks to increase the quality of logging, because that’s typically not a priority, not an explicit requirement and there are no accepted tropes on how to do this. Which is why I think that most popular stacks should come up with new patterns, in-app middleware layers and templates on how to achieve this. Stay tuned!