Enhancing Contextual User Explainability of Neural Network Predictions
Background
Neural networks are some of the most popular machine learning algorithms used today. This can be attributed to the fact that they boast unmatched predictive accuracy and a high speed of classification as compared to other algorithms [1]. However, the internal structures of neural networks are hidden and difficult to decipher, resulting in them often being referred to as “black boxes”. The inherent lack of transparency makes it a challenge to understand the reasoning behind the predictions and results obtained from neural networks.
This problem is particularly important to solve given how widely neural networks and other machine learning algorithms are being used today. In addition to making the algorithms difficult to debug, this property of neural networks potentially induces unwanted and unforeseen biases against certain groups of people. An example of this problem is the Apple Card fiasco in 2019, where Goldman Sachs came under fire when female customers were unexpectedly assigned lower credit limits than their male counterparts by the company’s algorithms [2]. The unwarranted bias associated with the algorithm eventually led to the company being investigated for gender discrimination.
To combat this problem, there is a growing desire for explainable artificial intelligence (XAI), as current systems have been described as being difficult for users to “understand, appropriately trust, and effectively manage” [3]. Basic XAI properties have progressed to become a necessity; the “right to explanation” within the General Data Protection Regulation (GDPR) in the European Union has made XAI a requirement for the continued use of AI systems in decision-making [4].
Motivated by this pressing need for XAI, an efficient framework is proposed in this article to improve the explainability of neural network results by providing simple post-hoc explanations for the results produced by the algorithm. The work described in this article builds upon the previously explained neural network implemented for cybersecurity purposes, details of which are available in the related article published on this Medium page [5].
Methodology
As a quick recap of the previous related blog post [5], a recurrent neural network was constructed and optimized for the purpose of detecting advanced persistent threats (APTs) using anomaly detection with Windows operating system event logs. The network was trained and tested on a subset of the publicly available “Comprehensive, Multi-Source Cybersecurity Events” dataset published by the Los Alamos National Laboratory [6].
In the context of this use case, an XAI-enhanced system will be useful in providing basic explanations of why specific logs are flagged as anomalous. The network that was detailed previously in [5] was only able to flag anomalous logs, but could not explain why the log was anomalous. The enhanced value that XAI can provide in addition to the previously employed neural network is summarised in orange in Figure 1 below, which is adapted from [3]. In the illustration, the anomalous nature of the log is justified by the URL contained in the log being malicious. The XAI addition provides useful additional information that can help human users better trust the system, and also helps in the debugging process when required.
The basic explanations shown in Figure 1 can be obtained by identifying the individual fields that are most responsible for the event being flagged. Following the same procedure laid out in the previous article [5], each log is first split into sequences of equal size, which are individually fed into the neural network. This process allowed for the calculation of each token’s individual contribution to the overall loss value of the log. An overview of the process is reproduced from [5] in Figure 2 below. Since the loss value is the penalty levied on the model for a bad prediction [7], the field that is most responsible for the log being identified as an anomaly can be determined as the token that gives the highest loss value.
After the high-loss field has been identified, norm-finding can be implemented to find the most appropriate replacement for the specific field. This is done by searching over the space of all possible token outputs for that field, and finding the replacement that maximally reduces the log’s overall loss value. By finding this norm, the system can thus not only inform the user of the problematic field, but also provide a reference for the user to compare against to check for disparities.
Testing Results
After testing with various input sequence sizes on the Intel DevCloud platform, it was determined that the optimal window size to use was 2. Combining this with the contextual knowledge of the log format, an updated overview of the loss calculation process is shown in Figure 3.
Given the contextualized process as seen in Figure 3, the tokens contributing the most to the overall loss value can be identified easily. For example, the fourth loss value L₄ can be attributed to the “Logon Type” field; if L₄ has the largest value, it is likely that the token for the “Logon Type” field in the log is highly anomalous, and is responsible for the log being flagged. Using the procedure described above, the dataset was tested and an example of the obtained results is illustrated in Table 1 below.
Taking the log in Table 1 as an example, the highlighted token for the “Source Computer” field contributed the most to the overall loss of the log. Through the norm-finding process, it was determined that a modified log with “C19497” as the token for the “Source Computer” field would have given a significantly lower individual loss value of 1.67. Using this token, the overall log loss value decreases from 43.3 to 16.3, representing an improvement of approximately 62%, and making the log no longer flagged as an anomaly.
From these results, it can be contextualized that the example event log has been flagged as the user is logging onto his account from a different computer, since the “Source Computer” field is highly anomalous. This problem could possibly be attributed to a stolen account. As compared to a traditional neural network system which determines anomalous logs but not the individual tokens responsible, the XAI-enhanced system thus provides a greater amount of information that can identify the probable cause of the cybersecurity incident.
Another subtle advantage of using loss values to determine anomalous logs is the ability to prioritize logs based on how urgently they need to be investigated. Since logs are flagged as anomalous when the observed outputs are out of the ordinary, a higher loss value indicates the presence of tokens that are increasingly rare. An example of this comparison is shown in Table 2 below.
Both logs in Table 2 share the same tokens other than the highlighted “Destination Computer” field, which is considered the high-loss field for both logs. The 36% higher loss value of “C108” in Log 2 as compared to “C625” in Log 1 suggests that the authentication event in Log 2 is a lesser-seen occurrence. This could signal an attempt by an APT to contact its sender after lying dormant after a long time, which is a key characteristic of the APTs that are being detected by the network [8]. Therefore, between the two logs in Table 2, users should prioritise the investigation of Log 2 as it is a more pressing problem to solve.
Conclusion
In this article, we have explored a framework to build explainable elements into neural networks, which gives additional contextual information to shed light on the generated predictions and results. While this is a significant first step, further work in this field will be critical to resolve the opaque property of neural networks, and help us understand the inner workings of these fascinating algorithms.
References
[1] Waltl, B., & Vogl, R. (February 2018). Explainable Artificial Intelligence — The New Frontier In Legal Informatics. Jusletter IT, 4:1–10.
[2] Nedlund, E. (12 November, 2019). Apple Card Is Accused Of Gender Bias. Here’s How That Can Happen. Retrieved from CNN Business: https://www.cnn.com/2019/11/12/business/apple-card-gender-bias/index.html.
[3] Gunning, D. (November 2017). Explainable Artificial Intelligence Program Update. Retrieved from DARPA: https://www.darpa.mil/attachments/XAIProgramUpdate.pdf
[4] Wachter, S., Mittelstadt, B., & Floridi, L. (2017). Transparent, Explainable, and Accountable AI For Robotics. Science Robotics, 2(6).
[5] Phee, B. (2019). Securing Windows: Machine Learning For Cybersecurity. Retrieved from Medium: https://medium.com/intel-student-ambassadors/securing-windows-machine-learning-for-cybersecurity-57835c34258e.
[6] Kent, A. (2015). Cybersecurity Data Sources For Dynamic Network Research. Dynamic Networks in Cybersecurity.
[7] Google. (n.d.). Descending Into ML: Training And Loss. Retrieved from Google: https://developers.google.com/machine-learning/crash-course/descending-into-ml/training-and-loss
[8] Thomas, R. (2016). A Little About Advanced Persistent Threats. Retrieved from SecVibe: https://secvibe.com/a-little-about-advanced-persistent-threats-4b7d40cab49.
