The ChatGPT security bug another zero day vulnerability hunger episode

The ChatGPT security bug is related to the Redis client library redis-py, which could allow a malicious Redis server to execute arbitrary code on the client system. After the first two hunger episodes OpenSSL, and Log4J the IT sector ashamedly present episode 3 redis-py.

A community of developers who contribute their time and expertise to improve the software typically maintains and found Open Source libraries. While many Open Source projects have dedicated maintainers who are responsible for reviewing code, fixing bugs, and releasing new versions, the reality is that a small group of volunteers who have limited resources and are stretched thin maintain many projects.

Additionally, many Open Source libraries are used in a wide range of applications and systems, making them critical components of the digital infrastructure. As a result, vulnerabilities in these libraries can have far-reaching consequences and may require significant resources to fix and mitigate.

The zero-day vulnerabilities in OpenSSL and Log4j illustrate the challenges of maintaining and securing Open Source software. While so many enterprise systems use these libraries and are critical to the functioning of many systems, only small groups of volunteers maintain these libraries and may not have had the resources or expertise to detect and fix vulnerabilities in a timely manner.

Furthermore, the Open Source community operates on a voluntary basis and is not always able to devote the necessary resources to identify and fix vulnerabilities quickly. In many cases, they only discovered vulnerabilities after attackers exploited them. At which point the damage has already been done.

To address these challenges, it is important for the Open Source community to prioritize security and invest in tools and processes that can help identify and mitigate vulnerabilities in a timely manner. This may include greater collaboration between developers and security researchers, more robust testing and quality assurance processes, and improved documentation and education around best practices for securing Open Source software.

Photo by Stormseeker on Unsplash

To avoid similar security bugs in the future, here are some general best practices:

Keep dependencies up-to-date

As software vulnerabilities are discovered, developers release patches and updates to address them. It’s important to keep all software and libraries up-to-date, including dependencies, to ensure that we address any security vulnerabilities.

Use only trusted libraries

When selecting third-party libraries or dependencies, it’s important to verify that they are widely used, maintained, and have an excellent reputation. Avoid using libraries that are not actively maintained or have a history of vulnerabilities. GitHub has an excellent feature showing the maintainers and contributors.

Insights of the redis-py maintainers

Insights of the OpenSSL maintainers

Insights of the log4j-2 maintainers

Validate inputs

When accepting inputs from external sources, such as user input or data from a network connection, it’s important to validate and sanitize the input before using it. This can prevent common attacks, such as SQL injection and cross-site scripting.

Follow secure coding practices

Developers should follow secure coding practices, such as input validation, output encoding, and password hashing, to prevent common security vulnerabilities.

Conduct regular security audits

Regular security audits can help identify potential vulnerabilities in software and libraries. These audits should include both automated scans and manual testing by experienced security professionals.

Have a vulnerability reporting and response process in place

It’s important to have a clear and well-documented process for reporting and responding to vulnerabilities in software and libraries. This should include a mechanism for users to report vulnerabilities, a process for verifying and addressing reported vulnerabilities, and a plan for releasing patches and updates in a timely manner.

Fight security related bugs

To maintain and fund a library, we can take the following steps to help reduce the risk of security bugs:

Regular code review

Experienced developers should regularly review the library to identify potential security vulnerabilities and other issues. This review can include both automated scans and manual testing.

Security-focused testing

The library should undergo comprehensive security-focused testing to identify any vulnerabilities or weaknesses that may be present. This testing should include both unit tests and functional tests.

Promptly address reported vulnerabilities

If we report a vulnerability in the library, it should be directly addressed and a patch or update released as soon as possible. We should establish a simple process for reporting vulnerabilities and addressing them in a timely manner.

Documentation and education

The library documentation should include guidance on best practices for using the library securely, such as how to handle sensitive data and how to avoid common security pitfalls. Additionally, education and training should be provided to users of the library to ensure they understand these best practices.

Long-term maintenance and support

To ensure the library continues to be maintained and updated over time, a plan for long-term maintenance and support should be established. This may involve funding and resources to ensure we keep the library up-to-date and we address promptly any security vulnerabilities.

Collaborate with the security community

The library maintainers should collaborate with the security community, including security researchers and other developers, to identify potential vulnerabilities and improve the security of the library. This may involve participating in bug bounty programs or engaging with security researchers directly.

Reviewing the ChatGPT security bug

Setting up your dedicated Redis instance on Linux using snap. Snap is such a breeze, so that the following command install and start the most current Redis server in our latest Ubuntu developer environments.

sudo snap install redis
sudo snap start redis

Connect to the running redis instance using the redis command-line interface.

/snap/redis/current/usr/bin/redis-cli

Setup a dedicated conda environment.

conda create -n redis-smoke
conda activate redis-smoke
conda install -c conda-forge redis-py

For reproducing the security bug, we use the asyncio module to create an asynchronous Redis client using the redis.asyncio library. The code directly connects to the local instance, usually you set up a Redis connection with a specified host and password, and with SSL enabled. It then sets two Redis keys (company and buggy) and retrieves their values using the get method.

The code also demonstrates how to cancel a Redis command that is executing asynchronously using asyncio.create_task() and asyncio.sleep(). Specifically, it creates a task to retrieve the value of the company key and then cancels the task after a short sleep. If the task is not canceled in time, the code prints an error message. Otherwise, it prints a message indicating that we successfully canceled the task.

import asyncio
from redis.asyncio import Redis



async def maintain_async():    
    async with Redis(host='localhost', port=6379, db=0) as redis_client:

        await redis_client.set('company', 'openAI')
        await redis_client.set('buggy', 'redis-py')

        new_task = asyncio.create_task(redis_client.get('company'))
        await asyncio.sleep(0.001)
        new_task.cancel()
        try:
            await new_task
            print('try again, we did not cancel the task in time')
        except asyncio.CancelledError:
            print('managed to cancel the task, connection is left open with unread response')

        print('buggy:', await redis_client.get('buggy'))
        print('ping:', await redis_client.ping())
        print('company:', await redis_client.get('company'))

if __name__ == '__main__':
    asyncio.run(maintain_async())

Running this implementation directly in the developer environment where the redis server is installed, we were not able to force the cancelled error, leaving the connection in an open, unsafe state with an unread response. The loop-back is a virtual interface that is always up and reachable as long as at least one of the IP interfaces on the switch is operational. So that the task result is available after a few milliseconds.

But when we setup the redis server in one of our cloud-based developer environments, and run our smoke tests using a dedicated build pipeline, so that the client needs to send data over the network interface such that requests typically take more than 10ms, we immediately experienced the cancelled errors raising up.

We are far from perfect, some potential bugs or defects in this smoke test we did not address:

Improper handling of exceptions

The code only catches and handles asyncio.CancelledError. However, other exceptions could be raised during execution, such as redis.exceptions.RedisError, which should be caught and handled appropriately.

No error handling for Redis commands

The code sets Redis keys and retrieves their values using the set and get methods, respectively, but it does not handle any errors that might occur during these operations, such as redis.exceptions.ConnectionError or redis.exceptions.ResponseError.

Inconsistent use of await

In the maintain_async function, some Redis commands are awaited (await redis_client.set(...)) while others are not (print('buggy:', await redis_client.get('buggy'))). Consistent use of await is recommended for readability and to ensure that asynchronous operations complete before continuing with subsequent code.

Lack of input validation

The code assumes that the correct number of arguments are passed in via the command line, but does not validate or sanitize the input. This could lead to unexpected behavior if it provided incorrect or malicious input.

Discussion

Developers play a critical role in ensuring that the software they create is secure and free of vulnerabilities that attackers could exploit. Security-related bugs can have significant consequences, including data breaches, financial losses, and damage to reputation. As such, it’s important for developers to take security seriously and take steps to minimize the risk of security-related bugs.

One way to help reduce the risk of security-related bugs is to use compiled languages like C++ and Rust, rather than scripting languages. While scripting languages like Python and JavaScript can be convenient for rapid prototyping and development, they may also be more prone to vulnerabilities such as buffer overflows and code injection attacks. Compiled languages provide stronger type checking and memory management capabilities that can help prevent these types of vulnerabilities.

Using compiled languages can also help ensure that code is optimized for performance, which is critical in applications where speed, carbon-effectiveness, and static security checks are important, such as high-frequency microservices and system-to-system communications.

In conclusion, by taking care of security-related bugs, collaborating with security experts, and also using compiled languages like C++ and Rust, developers can help improve the security and performance of their applications, reduce the risk of vulnerabilities, and protect the data and reputation of their organizations.

Let’s work together to create secure and carbon-effective software that meets the needs of today’s users and businesses.

References

[1] Off by 1
Canceling async Redis command leaves connection open, in unsafe state for future commands.