Intrusion Detection is Failing
Every day, we hear of a new security breach. Data is plundered, user credentials exposed, and corporate secrets are compromised. Millions of user records, and terabytes of data are exfiltrated and sold on the open market. Although many of these compromises are due to poor security practices, others are the result of Advanced Persistent Threats that continue for days, if not weeks. They can cause untold damage.
Intrusion Detection is intended to be an effective defense against these more sustained attacks. Intrusion Detection Systems should detect such penetrations, and alert us to the threat. However, many IDS solutions are being overwhelmed and bypassed relatively easily. In the cloud, Intrusion Detection has struggled to adapt to the lack of a well defined network perimeter and highly dynamic nature of the cloud. So Intrusion Detection is failing badly, and in the war to secure systems against the onslaught, it is falling further behind each day.
The Security Landscape
Intrusion Detection Systems have a long history and were first implemented in the mid-eighties. Not surprisingly, the security landscape has changed dramatically with mass migration to the cloud. Further, over the past decade, the volume of attacks has increased dramatically (up 36 percent in 2015), and the sophistication of these attacks is escalating. Complex hacks and techniques are being weaponized, packaged as toolkits and made available to a large cast of characters. This is fueling a growing and extensive cybercriminal ecosystem. Unfortunately, our protections have not evolved at the same pace as the rapidly changing threat landscape.
The Cloud Changes Everything
The cloud has a very different threat profile to the enterprise on-premises data center. The cloud is highly dynamic with computing assets and other resources being rapidly and frequently allocated and destroyed. Cloud configuration is highly malleable and responsive to changing requirements and many organizations are scaling their use of the cloud on a minute by minute basis. At its core, the cloud has a much less defined network perimeter than its enterprise counterpart, and this makes network based intrusion detection much less effective. Consequently, a security rethink is required for the cloud and simply applying enterprise Intrusion Detection Systems to the cloud is not adequately addressing the security need.
What are Intrusion Detection Systems?
Intrusion Detection Systems (IDS) are a primary defensive capability against cyberattacks. They monitor networks and systems for malicious activity and are are typically classified as either Network Intrusion Detection Systems (NIDS) or Host-based Intrusion Detection Systems (HIDS). Internally, IDS use two primary detection techniques: signature-based detection where malicious traffic, data or programs are identified by their pattern or signature, and anomaly-based detection where a system’s current behavior and access patterns are observed allowing deviations from the expected to be detected and reported.
It is important to realize that signature-based approaches are reliant on capturing and codifying each exploit, so that a unique identifying signature can be created for the specific attack. Only then, can the signature be deployed to other systems to protect them as well. This is required for each and every attack. More about this later.
The problem with anomaly-based detection is that in a diverse network, it becomes very difficult to distinguish a malicious pattern or behavior from a friendly one. Indeed, anomaly-based detection is less effective the further the IDS is removed from the actual asset in need of protection. This places real limits on how effective an anomaly-based NIDS can be.
IDS are Bypassed
Clearly, intrusion detection is critically important. However, it is relatively easy for a talented attacker to bypass these systems. Increasingly, these techniques are being “weaponized” into convenient tool kits for attackers who cannot develop their own.
Bypassing Signatures
Signature-based IDS detect specific sequences of bytes or instructions from malware to identify malicious behavior. However, it is relatively straight-forward for attackers to slightly permute the pattern of malware bytes and thus evade signature detection. Signature-based IDS catch well understood, high-volume malware varieties, but completely miss the more tailored or targeted malware derivatives. Consequently, signature-based IDS are not effective against a talented, and careful cyberattacker.
There is also a window of vulnerability from the time malware is released to when an appropriate signature can be crafted and deployed. It may take days, weeks, or many months while systems are exposed without a signature to catch even the vanilla variety of the malware.
Although a signature-based NIDS certainly adds value by sanitizing the network stream, relying solely on a NIDS to eliminate all threats is an unreasonable expectation. Signature-based approaches will always fail to detect and prevent new malware derivatives.
Bypassing Anomaly Based Detectors
Anomaly-based intrusion detection has its own problems as well. Some security vendors make grand promises and assert that they can observe the slightest change in network traffic and detect attackers moving through the network. In reality, the further the NIDS is removed from the actual endpoint assets, the less able it is to detect anomalous traffic. When you average the traffic of many different clients and servers over a common network, the extensive variety of network connections, destinations, and traffic provide substantial cover for an attacker.
To use an analogy, think of a single person and imagine what would be regarded as unusual behavior for that specific person. Now think of an entire city of people, say New York. Detecting unusual behavior for a city is much more difficult because of the diversity of its inhabitants. Spotting anomalous behavior is very difficult because someone, somewhere will exhibit that behavior in their normal routine. It is the same for anomaly-based network intrusion detection. The more assets that are behind the NIDS, the less effective it will be in determining behavior unusual for one specific application or server.
Talented attackers will utilize this difficulty and disguise their malicious traffic to blend in with the legitimate stream of network traffic. They will wrap their attack code inside what appears to be valid traffic and will vary the time and origin of their network packets. Probes and attacks can also be spaced over a longer period of time to make correlation difficult. Confronted by attacks using these techniques, the NIDS will struggle to determine whether the brief pulses of activity are malicious or not.
A better approach is to move the IDS closer to the asset and detect changes in unique behavior for the specific endpoint application or system.
Network-IDS are typically focused on external threats; however, the more sinister attacks may come from inside the network. In the past, the network perimeter was well defined so that securing the few simple network access points seemed plausible. Today, especially in the cloud, networks are overlapping, porous highways of trusted and untrusted traffic. Instead of the simple binary internal/external dichotomy, we grant access to employees, partners and application services that span the globe. The simple network perimeter is no more. The belief that you can secure the perimeter and defend the assets behind the firewall is outmoded and dangerous.
Instead of regarding the network as a safe zone of secure, pure traffic, it is better to think of it as a hurly-burly mix of friendlies and unfriendlies, good and bad traffic. With this understanding, individual assets should be protected without relying on a “superman” NIDS.
IDS are Overwhelmed
Today, the sheer number and variety of exploits is exploding. In 2015 Symantec saw over 430 million new, unique pieces of malware, up 36% from the year before. Detecting, diagnosing, creating and deploying signatures for this volume of attacks is an untenable task. Every dimension of the attack profile is increasing: the number of attackers, the volume of attacks and the variety of attacks are all increasing. It is impossible for security companies and IDS to keep up with this pace. To make matters worse, currently deployed IDS signatures are increasingly out of date and the exposure risk is increasing.
IDS Users are Overwhelmed
Under this torrent of malware and attacks, existing IDS are performing poorly. Much of this malware forms an adverse background that does not require a foreground response. For example: SQL injection attacks can be mitigated by using prepared statements and stored procedures. With these protections in place, alerts of a potential SQL injection are unhelpful at best. However, traditional IDS will generate alerts for each and every potential threat creating a deluge of alerts that overwhelms security staff. This “crushing weight” of alerts obscures real threats and exhausts support staff. This is especially true of signature-based NIDS that cannot understand the specific needs or environment of each asset behind the appliance.
When NIDS alert on each and every piece of malware or anomalous traffic, many false positives are generated. False positives are also generated by simple routine tasks such as O/S updates and configuration changes. Many sites report hundreds of new alerts being generated per minute. The sheer volume delays timely response to real threats as support staff sift through clogged alert queues. At these levels, security staff cannot devote their time to real threats and attacks underway. This load effectively “trains” support staff to ignore the alert queue at their own peril.
Furthermore, when the NIDS is relatively far from the assets being secured, the actual severity of the threat and context of the attack is poorly captured. This means that important threats are costly to investigate and triage as there is often important information missing from the alert details.
IDS are Costly to Install
In an attempt to manage this flood of alerts, IDS vendors provide the means to tune their products to dispatch and/or suppress unwanted data. As the security support staff dispatch alerts, they tune the IDS to their specific, current system and network patterns. This can take several weeks of work to stem the tide of alerts. Thereafter, any significant change of the network or system will require a re-tune and the work must be repeated. Companies without extensive security support staff to tune their IDS, typically find the out-of-the-box IDS configuration to be far too noisy, and often, they simply ignore the alerts and the IDS in general. General IDS solutions can be very costly to install and maintain due to this configuration requirement. Consequently, IDS are mostly used by larger organizations that can afford the staff to manage the IDS.
The Road Forward
For the cloud and in the prevailing landscape of increasing cyber threat, it is essential to address the shortfalls of current IDS offerings. In summary, IDS suffer from:
- being tedious and difficult to install, configure and tune.
- generating a large (crushing) volume of poorly qualified alerts.
- failing to detect important indicators of compromise that are only available inside servers or containers.
- delivering poor performance using too much CPU and memory resource.
- high installation and maintenance costs.
- being a vulnerability themselves as they are prone to attack, bypass and compromise.
At SenseDeep, we believe that everyone should be able to easily and cost-effectively defend their cloud-based systems. Improved intrusion detection is essential for this, but a new approach is probably needed.
About SenseDeep
SenseDeep is a Seattle based startup dedicated to providing effective cloud management for DevOps. It has released the PowerDown cloud management platform to easily lower cloud costs.
References
Lower Cloud Costs for DevOps. PowerDown idle cloud resources.
