Uploading files to AWS S3 using Axum — a Rust Framework

image from adamtheautomator.com

In this article, we are going to create an API using the Axum Rust Programming Framework to upload files to Amazon S3. Our attempt is to make this a generic API. We welcome you to provide suggestions to further enhance the same.

Do check the code on GitHub. Please look at the environment variable section or the README.md file for using the code.

Technologies used -

• Axum web framework { For creating API }
• AWS S3 (Simple Storage Service) { File is uploaded here }

First, go to any folder where you want to code and then run the following command to create a rust project.

cd rust_projects
cargo new aws-s3-file-upload-api-rust

Dependencies/crates -

In your Cargo.toml file add the following dependencies

# Cargo.toml

[package]
name = "aws-s3-file-upload-api-rust"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
axum = { version = "0.5.17", features = ["multipart"] }
serde_json = { version = "1.0.68" }
tokio = { version = "1.0", features = ["full"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tower-http = { version = "0.3.0", features = ["cors"] }
chrono = { version = "0.4.22", features = ["serde"] }
aws-config = "0.51.0"
aws-sdk-s3 = "0.21.0"

We are using multipart feature of axum, since we are going to send files as form data multipart, serde_json for parsing JSON data and sending JSON responses, tokiofor async runtime (used by axum ), tracing and tracing-subscriber for logging, tower-http for configuring CORS (so that we can use the API in browsers), chrono for timestamp (we will get into it in the article), aws-config for handling AWS configurations like credentials, region, etc and aws-sdk-s3 SDK for interaction with AWS S3 as we can guess from the name itself.

Setup Up API

In the src/main.rs file import the following modules and types

// main.rs

use axum::{
    extract::Multipart,
    http::StatusCode,
    routing::{get, post},
    Extension, Json, Router,
};
use std::collections::HashMap;
use tower_http::cors::CorsLayer;
use tracing_subscriber::{prelude::__tracing_subscriber_SubscriberExt, util::SubscriberInitExt};

use aws_sdk_s3 as s3;

use s3::Client;

then paste the following code below the imports, it’s the main function with all the configurations that spins our API server

// src/main.rs

#[tokio::main]
async fn main() {

    // configuration logging and initiate it
    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::new(
            std::env::var("RUST_LOG").unwrap_or_else(|_| "aws-s3-file-upload-api-rust=debug".into()),
        ))
        .with(tracing_subscriber::fmt::layer())
        .init();

    // configure your cors setting
    let cors_layer = CorsLayer::permissive();

    // the aws credentials from environment
    let aws_configuration = aws_config::load_from_env().await;

    //create aws s3 client
    let aws_s3_client = s3::Client::new(&aws_configuration);

    let app = Router::new()

        // route for testing if api is running correctly
        .route("/", get(|| async move { "welcome to Image upload api" }))

        //route for uploading image or any file
        .route("/upload", post(upload_image))

        // set your cors config
        .layer(cors_layer)

        // pass the aws s3 client to route handler
        .layer(Extension(aws_s3_client));
    let addr = std::net::SocketAddr::from(([0, 0, 0, 0], 3000));
    tracing::debug!("starting server on port: {}", addr.port());
    axum::Server::bind(&addr)
        .serve(app.into_make_service())
        .await
        .expect("failed to start server");
}

I have commented out the code but let me explain it briefly

First, we are configuring the logging, here if you have used a different name for your project replace aws-s3-file-upload-rust=debug with you-project-name=debug or else you won’t see any logs, then we are configuring the CORS I am keeping it permissive so anyone can access the API in the browser but you can configure it as you want check here, then we are getting the credentials for AWS from the environment, the aws-config crate tries to get env variables from many places you can check here, then create a client for AWS S3.

After configuring everything pass it to the route constructor, then start the server, if you have used axum before then you will know this part, if not then you can refer to my previous articles on Axum.

Note: the root endpoint / is for testing if the API is running properly also the handler for /upload , we will create in the next section.

Creating the handler for uploading files

In the same file src/main.rs copy and paste the handler code

Note: it might look long but believe me it’s not and it’s really not complicated, it’s just formatted well I would say

// handler to upload image or file
async fn upload_image(
    Extension(s3_client): Extension<Client>,
    mut files: Multipart,
) -> Result<Json<serde_json::Value>, (StatusCode, Json<serde_json::Value>)> {
    // get the name of aws bucket from env variable
    let bucket = std::env::var("AWS_S3_BUCKET").unwrap_or("my-bucket-name".to_owned());
    // if you have a public url for your bucket, place it as ENV variable BUCKET_URL    
    //get the public url for aws bucket
    let bucket_url = std::env::var("BUCKET_URL").unwrap_or(bucket.to_owned())  
    // we are going to store the respose in HashMap as filename: url => key: value
    let mut res = HashMap::new();
    while let Some(file) = files.next_field().await.unwrap() {
        // this is the name which is sent in formdata from frontend or whoever called the api, i am
        // using it as category, we can get the filename from file data
        let category = file.name().unwrap().to_string();
        // name of the file with extention
        let name = file.file_name().unwrap().to_string();
        // file data
        let data = file.bytes().await.unwrap();
        // the path of file to store on aws s3 with file name and extention
        // timestamp_category_filename => 14-12-2022_01:01:01_customer_somecustomer.jpg
        let key = format!(
            "images/{}_{}_{}",
            chrono::Utc::now().format("%d-%m-%Y_%H:%M:%S"),
            &category,
            &name
        );

        // send Putobject request to aws s3
        let _resp = s3_client
            .put_object()
            .bucket(&bucket)
            .key(&key)
            .body(data.into())
            .send()
            .await
            .map_err(|err| {
                dbg!(err);
                (
                    StatusCode::INTERNAL_SERVER_ERROR,
                    Json(serde_json::json!({"err": "an error occured during image upload"})),
                )
            })?;
        dbg!(_resp);
        res.insert(
            // concatinating name and category so even if the filenames are same it will not
            // conflict
            format!("{}_{}", &name, &category),
            format!(
                "{}/{}",
                bucket_url,
                key
            ),
        );
    }
    // send the urls in response
    Ok(Json(serde_json::json!(res)))
}

Please follow along the code for the below explaination

In the upload_image function (it can be used to upload any file, not just an image) we are extracting the s3_client which we configured in main function, we are going to use it to query the AWS S3 service, files are the files which are sent with the request in form data.

Result<Json<serde_json::Value>,(StatusCode, Json<serde_json::Value>)> this tells the function that the response will be JSON on success or JSON with some status code on error.

We are getting the name of the AWS S3 bucket from an environment variable AWS_S3_BUCKET .

We are initializing a variable res for storing the response that we are going to send on success. here we are using HashMap , which will be converted to JSON using serde_json::json! a macro when sending the response.

We are looping over every file which is sent in the form data (key-value pair) then we are getting the category (this is the name field or key in form data), filename (we can extract it from file data), and of course the data itself.

Then, we are creating a key, which is the path with the file name to store in AWS S3 I have a hardcoded images/ path but you can use any path in its place, you can even use the category (that’s why I named it category) which I extracted before. then for the file name, I concatenated the timestamp, category, and file name to make it unique.

Then, we send the upload request (putObject) to AWS S3 with the help of s3_client with all the above data.

In the res.insert() call we are setting the bucket URL for the respective image and then sending this res object as a response in the end.

For the bucket URL, you have to make the bucket public and change ACL (Access Control List) settings for the bucket, you can also set the URL to your own domain using Cloudfront. I am not an expert in the field of DevOps / infrastructures you can find other articles about it.

Running the API

First we are going to set the environment variables

Setting the environment variables

setting environment variables is different for the different shells, I use WLS2 for development, so sometimes I use bash, sometimes fish, and in windows I use PowerShell. To be honest I have to search the web most of the time for setting the environment variable, especially for the fish shell. if you are using bash you can use the export command.

Note: I use the last method to run programs with environment variables also i do recommend reading the README file at github

# inside the root of project
# set environment variables

export AWS_ACCESS_KEY_ID=your_aws_id
export AWS_SECRET_ACCESS_KEY=your_aws_secret_key
export AWS_REGION=aws_region
export AWS_S3_BUCKET=s3_bucket_name_(image/file will be uploaded here)
export BUCKET_URL=(if you have a public url for your s3 bucket)

# run the API
cargo run 

you can also run the program in a single line like below

# ya it's single line

AWS_ACCESS_KEY_ID=your_aws_id \
AWS_SECRET_ACCESS_KEY=your_aws_secret_key \
AWS_REGION=aws_region \
AWS_S3_BUCKET=s3_bucket_name \
BUCKET_URL=public_url_for_s3_bucket_optional \
cargo run

Testing the API

For testing, you can use any CLI app like curl , httpie , curlie , etc, or any GUI app like postman or the one I am using here httpie (it has both CLI and GUI versions).

Conclusion

It feels like a long article just to upload files 😅.

Thanks for reading the article, and I hope that it helps you in your projects as well as an introductory program using Rust.