What Would Useful ICS-CERT Stats Look Like?
Last week I wrote about the numerous problems with ICS-CERT’s annual stats. It’s easy to point out problems, but what stats would be useful and possible given ICS-CERT’s current and reasonably collectable data?
Here is my incomplete list.
Education and Awareness Statistics
Internet Connected ICS in a Country
This is an easy number to get from Shodan, and John Matherly said at KIACS that he is seeing linear growth in Internet connected ICS. These are not IoT devices; these are classic ICS devices using ICS protocols. This number should be going down, not up, if awareness activities are effective.
This number needs to be carefully presented because it almost certainly doesn’t represent the trend of critical infrastructure or high value ICS that are Internet connected. And ICS-CERT shouldn’t be taking the time to chase down and remove these one by one. However as my friend Eireann Leverett always points out, these ICS are likely important to some company or organization if not to the nation.
I think this is the best trend to track for broad based ICS security awareness in a country.
This is an online ICS security evaluation tool. A review of the pro’s and con’s of this tool is a different and long article. As noted in last week’s article the trend line of downloads of CSET is a credible statistic on awareness and effort in securing ICS, and as well as a credible statistic on ICS-CERT program effectiveness.
The statistics around CSET can be improved, and who better than Marty Edwards, Director of ICS-CERT, to explain how to improve CSET metrics in this 26 second excerpt from my S4x16 interview with Marty. He suggested tracking updates and continued use.
The raw data and trends presented in the vulnerability statistics are meaningless for many reasons, as explained in the previous article. This is unlikely to change in the next three years.
Vendor Vulnerability Handling
ICS-CERT could track the vulnerability handling capabilities of ICS vendors, either vendors they deal with in coordinated disclosure or a list of the key vendors in each critical infrastructure area. Some items to track:
- What percentage of ICS vendors have easily accessible contact information for reporting vulnerabilities?
- What percentage of ICS vendors have a secure method for a researcher to report the vulnerability?
- How long did it take for the ICS vendor to respond to the initial contact through the documented channel to a vulnerability issue?
- Key Item … how long did it take the vendor to provide an effective remediation for the vulnerability? It would be a plus if ICS-CERT could segment the vulnerabilities and remediation based on their relation to critical infrastructure and high value ICS.
Asset Owner Assessments
This is another area Marty and I explored in our S4x16 interview. It makes no sense for ICS-CERT to provide this service in order to secure ICS. They would need to scale it up by a factor of 100x or 1000x if securing ICS is the goal.
Or alternately they could perform these assessments on a small number of the most critical ICS in the country, assuming they have a prioritized list. The difficulty with this is ICS-CERT has to be invited to perform the assessment, and most of the asset owners with the highest priority ICS are unlikely to invite ICS-CERT in.
The only reason to perform these free assessments is to get and report a profile of the current security posture of a representative sample of a sector and track trend lines over the years. Getting this free assessment service from ICS-CERT should require acceptance that ICS-CERT will anonymize the information and report it out in aggregate per sector. I’d also recommend assessments only be performed on ICS where the asset owner commits to an annual update assessment so security posture improvement can be tracked.
This would require ICS-CERT to develop an assessment methodology and reporting that supports anonymized metrics. This isn’t difficult and could be based around the NIST Cybersecurity Framework. It would also require getting enough of the right participants from a sector to have a reasonable sample.
ICS Vendor Security Controls
ICS-CERT should have a list of the key ICS products for each critical infrastructure sector. I assume this list exists, and if it does not it is not difficult to create. The sectors tend to have a small number of vendors that represent most of the deployments (power plants: Emerson Ovation, Foxboro, GE … refining: Honeywell, Emerson Delta V, Yokogawa) and there are a small number of solutions that are widely deployed across all sectors (OSIsoft’s PI and some OPC servers).
These critical infrastructures ICS solutions should have key security controls and have removed common ICS security issues. For example, signed code, secure deployment guide, role based access control, no hardcoded credentials, patch testing and certification, authenticated protocol, support for application whitelisting, two-factor authentication … We can argue over what should be on this list, and the list should get longer over time.
It would not be difficult or time consuming to create a report card of how the identified high impact ICS solutions that are available for purchase are meeting this list of security requirements.
This is important because we need to stop digging the ICS security hole deeper with greenfield and major upgrade projects. If asset owners are stuck choosing between insecure by design offerings we will see new insecure deployments. In addition, asset owners could use this ICS vendor report card as part of their security evaluation in the next RFP.
ICS-CERT will not be able to provide useful statistical information on threat based on incidents addressed in the next three years. The numbers are too small and the data has a large collection bias. About the only valid information that can be drawn from this data is related to asset owner awareness of ICS-CERT and their willingness to share information with ICS-CERT.
If ICS-CERT insists on reporting this information then they need to better describe and categorize the incident and their activities in the response. At at minimum they should identify which incidents actually had an impact on the ICS. A large percentage of the “incidents” in the population only affect the corporate network.
I’d like to see statistics on the impact of the incident in terms of loss of production, waste, safety, etc. I’d like to see a breakdown of whether the attack had any ICS intelligence or was strictly using typical IT exploits and post exploit TTP. Again, this data cannot be used to quantify or characterize the threat, and simply reporting the activity is one of those counting stats that Marty was not happy with.
Literally ten years ago I spoke with ICS-CERT and asked why they continued to perform free ICS security training, and today it makes a lot less sense. SANS, ISA, and a variety of other organizations have quality ICS security classes providing in person and online.
There are only two reasons ICS-CERT should be performing this training:
- They have some unique training that is not widely available through other training courses. This is not currently the case. The other training options have surpassed what ICS-CERT offers.
- They are prepared and committed to train 10,000x or 100,000x more people than the 1,292 students they trained in 2016. This would have to include significant outreach to the asset owners to agree to train up their workforce.
ICS-CERT needs to get out of the training business. There are plenty of other tasks where they can play a unique and important role.
What statistics would you add to this list for ICS-CERT, or your national CERT, to track and report on? Remember it needs to be from currently available and reasonably collected data.