One of my former engineers, was tasked to evaluate several end-point protection products. Despite advising him to look into MISP & Cuckoo sandbox, he still insisted to do things manually...
Testing manually is unscalable:
- As sample size grows, files will be duplicated all over the place/machines.
- Keeping the information of each sample in spreadsheets is not scalable.
- Consolidating test results is tedious & error-prone.
- Trying to dig out old results of samples is painful without proper knowledge-management.
- Everyone just want to script it in his/her own way but nobody wants to document.
- Takes a long time to test a bulk of samples/offensive-techniques one-by-one. Consider the fact that each payload is to run within multiple target-Virtual-Machines installed with different Products-under-Test, revert VM to pre-test state… but I guess some people just prefer to take their own sweet time to do things.
After he left my team, I decided to do it myself. I just took it as an opportunity to get acquainted with Python & PyMISP.
The manual effort of preparing the PuT within targets still remains, but with APTC, at least the selected payloads are loaded into the targets & executed automatically & results can be consolidated with one-single-click of a tag:
The demo is a simple example of just dropping a payload (malicious Word file to exploit CVE 2017–0199) into the user’s desktop & launch. This test-case has 2 targets x 2 payloads. It can automate test runs beyond two targets & payloads. After each run, the VMs can be rolled-back via libvirt.
One can also use tools like AutoIT or Sikuli to launch GUI payloads. It is also not limited to product testing, one can use PEC to stuff like Application Whitelisting evasions. The target need NOT be VMs, it can be physical machines or devices or submit the payloads to detection/analysis engines (eg. Cuckoo Sandbox).
Details: https://jymcheong.github.io/aptc/
Source codes:https://github.com/jymcheong/aptc
