CVE Binary Tool
The CVE Binary Tool scans for a number of common, vulnerable open source components such as OpenSSL, libpng, libxml2, and expat to let you know if a given directory or a binary file includes common libraries with known vulnerabilities known as CVEs(Common Vulnerabilities and Exposures). It is also available as a python package.
This year my proposal was selected for Improving Triage and Tracking and enabling Incremental Updates as a part of Google Summer of Code, 2021 for CVE Binary Tool.
Organisation — Python Software Foundation
Sub-Organisation — CVE Binary Tool
Project — Improving Triage and Tracking & Enable Incremental Updates
Improving Triage & Tracking
The CVE Binary Tool can triage scans to mark issues as false positive, fixed, or otherwise keep notes as part of the scan. This issue covers some possible improvements, including the ability to merge scans and triages and helping projects track time to fix vulnerabilities.
Goal:
- Allow saving the output in the form of an intermediate report which can later be merged with other reports of the same format.
See the detailed guide on intermediate reports for more information. - Allow filtering triage on the basis of tag.
I have added intermediate reports which can be generated using the --append -a argument. You can also specify a tag using --tag -t which can be later used for filtering out intermediate reports.
Create an intermediate report using:
python -m cve_bin_tool.cli -a /reports_path/backend.json -t backend /home/backend/Here we are saving the intermediate report in /reports/backend.json.You can also omit the intermediate report path. CVE-Binary Tool will generate a filename with the default naming convention which is: "append.YYYY-MM-DD.hh-mm-ss.json"
python -m cve_bin_tool.cli -a /reports_path/ -t frontend /home/frontend/Multiple intermediate reports can be merged using the --merge -margument.
python -m cve_bin_tool.cli -m /reports_path/backend.json,/reports_path/append.2021-08-18.00-00-30.jsonThe output structure for the intermediate report is:
{
"metadata": {
"timestamp": "2021-08-18.00-00-30",
"tag": "backend",
"scanned_dir": "/home/backend/",
"products_with_cve": 139,
"products_without_cve": 2,
"total_files": 49
},
"report": [
{
"vendor": "gnu",
"product": "gcc",
"version": "9.0.1",
"cve_number": "CVE-2019-15847",
"severity": "HIGH",
"score": "7.5",
"cvss_version": "3",
"paths": "/home/backend/glib.tar.gz,/home/backend/gcc.tar.gz",
"remarks": "NewFound",
"comments": ""
},
...
]
}Filter Intermediate reports using --filter -F argument. This will merge only those reports which contain the tag “backend” in the directory path /reports_path/
python -m cve_bin_tool.cli -f backend -m /reports_path/ The merged output from multiple intermediate reports can be used to compare the severity trace in the HTML and PDF output as well.
Intermediate Triage also allows us to compare the total files, and product with/without CVEs traces according to the increasing DateTime value.
NVD API & Incremental Updates
The CVE Binary Tool currently refreshes the database using JSON feeds available on NVD.
The CVE API is the next stage in providing up to date vulnerability information for NVD data consumers. The results from this API are updated as quickly as the NVD website itself (unlike the traditional feeds which have explicit update intervals of once per day).
The NVD REST API takes less bandwidth and is more reliable in comparison to the downloadable feeds(which can also be removed in the future)
It also takes less space as we don’t need to save the JSON in the cache directory.
Goal:
- Add the NVD REST API to CVE Binary Tool.
- Allow using Incremental Updates.
The complete documentation on NVD REST API is available here.
A major benefit of using this NVD API is incremental updates which means that users won’t have to download the complete feed again if they require the latest CVE entries from NVD.
You can use this NVD REST API by specifying --nvd -n argument.
python -m cve_bin_tool.cli -n apiWhat is Incremental Update?
With the help of this CVE API, we can fetch just the newly added/modified NVD data using the timestamp of your current local database copy. This will fetch only the CVE entries for which any vulnerability or product string was modified or published later than the above-mentioned timestamp. This can save users a lot of time and internet bandwidth.How to use Incremental Updates?
You can use the pre-existing-u latestparameter along with the-n api. This will simply update your local database and cache copy with the newly published and modified NVD entries.
python -m cve_bin_tool.cli -u latest -n apiPull Requests:
Status:
Complete. The above-mentioned goals have been achieved and each PR has been merged into the codebase.
Major Takeaways:
- Linting, Documentation & asyncio:
I learned a lot about linting in python and how it can help improve the code health and style. Thanks to the mentors, I learned about writing the sacred scriptures; documentation — of which I had no prior experience. I learned how to write concurrent code using asyncio which reduced the NVD REST API download time to 5% in comparison to single thread requests.
- Time Management:
My entire GSoC period has been very productive. I never felt stressed or burnout. I had enough time for my college and internship even after watching a lot of movies throughout the summer.
Weekly Blogs:
Detailed weekly description of tasks and work done can be found at:
Acknowledgement:
On a final note, I am extremely grateful to my mentors, Terri Oda, Anthony Harrison, John Andersen, Harmandeep Singh and Saurabh Khandelwal for all their guidance. I am thankful for their motivation which helped me in improving the quality of my code.
I am also thankful to fellow contributors Dmitry Volodin, Harsh and Suhail for reviewing my Pull requests.
Thanks to Google, Python Software Foundation and Intel for this great opportunity.
Feel free to connect with me on Linkedin and check out my Github profile.
