System Clock Skewed? Read this Post, Especially if You Don’t Have Time
“Time keeps on slippin’ slippin’ slippin’, into the future” — “Fly Like An Eagle”, The Steve Miller Band
Your web server’s system time can keep on slippin’ far into the future if Network Time Protocol (NTP) is not properly configured. Having accurate system time is critical for application logic, scheduled jobs, and of course logging. With logging, if the system time is off, log forensics and log correlation of security events across systems becomes a nightmare, if not impossible. Many software components on your servers rely on accurate system time, including the Signal Sciences agent. The agent uses the local system clock to generate timestamps when logging detections and other events. As a result, ensuring accurate system time on all your web servers is a must for robust security — and this is especially true for virtual machine based deployments.
Why do I call out virtual machines? Our agent has been deployed in numerous environments and on thousands of machines (bare metal, virtual machines, containers, IaaS, PaaS, and whatever else you might imagine). In new deployments, virtual machines without NTP configured stick out like a sore thumb since their system time is way off. We’ve seen ranges from tens of thousands to hundreds of thousands of seconds off into the future! This drastic time skew has a severe impact on security related events since the timestamps can be so significantly skewed. One example of impact is when presenting events in a dashboard that is time based, e.g. displaying all events within the last 6 hours. You won’t see the data if its timestamps are several hours in the future. Time skew can happen on any infrastructure, but if you are running virtualized infrastructure you’ll want to pay close attention.
In addition to NTP not being configured, there has been research published on attacks against the NTP protocol. Some of which result in shifting time on NTP clients. This research is published here, and referenced in the Ars Technica article below. Another threat consideration is a malicious insider, who could modify system time in attempts to hide events or manipulate time sensitive transactions.
“Watching” Time
How can we mitigate the risk of skewed timestamps, missing data, and possible attacks on system time? First, NTP is a beautiful thing. Make sure you have a consistent NTP implementation across all your severs. Second, periodically verify the system time on your servers is not skewed beyond an acceptable threshold. Fortunately for Signal Sciences customers, you already have visibility into your server time! Since time is such a crucial part of security, our agent provides visibility on your server clock skew. This is just one of the many metrics reported on by the agent, and it can be found via the UI by navigating to the agent’s graph page. On that page, scroll down towards the bottom to find the Agent Clock Skew graph.
If you’re not a Signal Sciences customer, don’t worry, keep reading! In the next section I’ve published two scripts, one is a proof-of-concept Bash shell script that can be used to help monitor system time across many servers.
Monitoring Clock Skew
Ok, visibility on clock skew — great! Pretty graph — great! But what if you are like many of our customers and have hundreds of agents deployed. That’s a lot of looking at pretty graphs to find hosts with clock skew — not great! Good news, Signal Sciences makes all data available via it’s API — great!!! This means you now have the ability to automate monitoring of clock skew across your web server farm.
With the API you can integrate with any monitoring system you may have, however, I’ve provided an example of checking clock skew with just two scripts. The first script, SigSciApiPy, handles authenticating and pulling agent metrics from the API. The second script is shown below, clock_skew.py, loops through the agent data looking for the value of host.clock_skew. Host.clock_skew is the median number of seconds that the particular agent’s clock is skewed compared to the time on our servers, over the past 5 minutes. So for example, if we have “host.clock_skew”: 30, that means that we saw a median skew of 30 seconds over the past 5 mins. The clock_skew.py script will print a warning or alert message to the console based on defined thresholds.
An example run of these scripts together looks like this:
./SigSci.py --agents > /tmp/sigsci-agents.json; ./clock_skew.py
WARNING: Clock skew on host1 is 4
ALERT: Clock skew for host2 is 5
WARNING: Clock skew on host3 is 4
WARNING: Clock skew on host4 is 3
ALERT: Clock skew for host5 is 5
ALERT: Clock skew for host6 is 6Note, if you do use clock_skew.py, you’ll want to set the WARN and ALERT threshold variables to values that make sense to you.
If you are not a Signal Sciences customer, below is a script that can achieve a similar result in monitoring for clock skew. If shell scripts are not your thing, there are many other tools that may be a better fit in your environment. For example, monitoring systems like Nagios or Zabbix, or configuration management tools such as Ansible, Chef, or Puppet. I’m sure there are other, and even better, ways to monitor for clock skew. If you have suggestions please add them in the comments section.
Conclusion
I think we can all agree that ensuring you have accurate system time across your servers is critical for security. Also, having the ability to monitor for time skew is equally as important. Signal Sciences gives you this ability via its API, and with a few simple scripts you can monitor time skew across all your web servers. Even if you are not a Signal Sciences customer you should choose a monitoring method that makes sense for you, and start monitoring. For Signal Sciences customers, I hope the example scripts I’ve provided help you to jump start your clock skew monitoring efforts in a… timely manner.
Signal Sciences is the industry’s first Web Protection Platform (WPP) providing both Next Generation WAF and RASP technologies. Signal Sciences WPP was built in response to our own frustrations of trying to use legacy application security approaches while enabling business initiatives like DevOps, cloud adoption, and CI/CD.
