“How do I know any of this is secure?”

You’re buying a “security” product. But how vulnerable is the product itself?

I need to be very clear here — this is a minimal outline of a discussion you probably want to have with your software vendor. It certainly applies to security software, but also to any software where you care about security in one form or another. Again — this outline is a starting point.

You should not assume that buying from a brand name software vendor implies that the “right things” are happening behind the scenes. I’ve been involved in an epic, Apollo-13-grade cleanup inside of a very reputable vendor. You have to look into the details for the specific product that you care about. This is one area where a small company can easily do as well as or better than a large one.

When we at Nukona were landing our 3rd marquee customer (think Fortune-50 size), I had to fly out and give a technical prezzo. The customer’s “security guy” was a key member of the audience. A very sharp guy, a very quick-on-his-feet guy, a used-to-work-for-Bruce-Schneier guy. The slide that saved my ass was titled “How do I know any of this is secure?” Here is what we (Nukona management) committed to in that slide:

We committed to hiring outside help (a security firm)

A software vendor needs to have an outside firm engaged. Be vary wary of the “we have an internal team that does this” response from a vendor. If you’re thinking of accepting that answer, then you had better plan on digging a lot deeper.

With an outside security firm, it is harder for corporate politics or budgets to degrade the security process. The outside firm will want to protect its reputation, and customer expectations of an externally-generated report will force the budget issue.

The security firm must evaluate the architecture

Pen-testing alone is not enough. The security firm must have taken the time to examine the overall architecture in the context of what “security” means to that particular product. This analysis costs money, and needs to be refreshed as the product evolves. It defines the context for the subsequent testing.

To do this properly, the security firm must have access to the code, to the design documents, and to any staff/engineers that it deems necessary. Engineering management must have incoculated the “they’re here to save our butts” attitude toward the security firm’s intrusions.

The security firm must pen-test the product

Using the material they’ve generated during the earlier assessment, the security firm must penetration-test the product. Penetration-test (aka “pen-test” means that they try to hack the product. The outside firm will not want to perform a shoddy, underfunded test and then have the product hacked in an embarrassing way. It is bad for their reputation. This is good for you.

The security firm must have evaluated the software vendor’s processes

A good security firm will want to know the flow from requirements to tested code, and see that the process reinforces the development of strong designs and code.

The software vendor must share hardcopy with you, the buyer

You’ll almost certainly be required to sign an NDA to get any details. The software vendor’s word isn’t good enough, and neither is a document that they’ve generated. You want to see something in writing that was generated by the security firm. It will not be exactly the same as the report provided to the software vendor, but it should include an unfixed vulnerability listing and scoring from both the earlier assessment and the later pen-testing.

Do not expect the vendor to share the details of vulnerabilities with you. They aren’t going to release a guide on how to hack their product. But you need to know that there aren’t gaping holes, and a reputable security firm won’t write-up a document that will lie about gaping holes. Keep in mind, any product will have some issues — nobody is perfect. Perfection is a red flag.

The software vendor must have a vulnerability management process

Most software products these days use a ton of 3rd-party code or libraries (aka “modules”). It is not unusual for a medium-complexity system to rely on more than 100 3rd-party modules. Vulnerabilities are constantly being uncovered in the 3rd-party code, and the software vendor must have a reliable process for:

• tracking 3rd-party modules and module versions used in the product,
• learning about vulnerabilities discovered in the 3rd-party modules,
• quickly evaluating the relevance and impact of these vulnerabilities on the product,
• releasing security patches as needed.

You should be able to get details on this process in writing, and you should also be able to get proof that the process is functioning.

The dance between the software and security vendors must be ongoing

For the best results, the external security firm are treated as trusted advisors by the product development team. There is budget for the firm’s time, and the Engineering team can informally (as well as formally) involve the firm at will to ensure that security is baked in.

Assessment updates and pen-testing need to happen periodically, but more importantly need to be synchronized with major changes to the product.

During that aforementioned “epic” cleanup experience I often asked “What do the other teams/BUs do for <insert-security-issue-here>”. The answer was often “nothing”. Everyone was well intentioned, but ensuring security is perpendicular to rapidly developing functionality. I’ve seen a few factors combine to create a big mess:

• There isn’t a corporate priority placed on ensuring that products aren’t vulnerable. A corporate mandate could make an outline like the one above part of a company-wide product release criteria.
• Pressure to release. This will never change, but it needs to be recognized that the revenue-driven hurry to ship drives tunnel vision. The avoidance of vulnerability is not to be found in that tunnel.
• Group culture. If the culture is closed or secretive, it is more likely that security issues get swept under the rug or deferred until the mythical “next major release”. An open culture where bad news early is welcome, where shit is eaten early, and with big bites — such a culture of attacking small problems is effective in preventing the big ones.

It’s quite possible that you’ll catch your software vendor flat-footed. But at the end of the day, if they really do have your best interests at heart, then they’ll engage and improve. And you will be able to sleep easier.