Posting chunked body requests with hyper in Rust lang

I’ve been writing more Rust and I really can’t stop myself now! I’m currently working on an utility (work in progress) which runs tests similar to speedtest.net from command line.

To test the Tx, I wanted to create bunch of threads doing HTTP POST requests but I also wanted to time it and shut it down as soon as possible so that I don’t have to hang on for a long time in poor connections just as implemented in this work that’s already been done. The basic idea is you send data in chunks so that you have a handle on when to stop sending.

I’ve been using hyper as HTTP client. Usual post requests when using hyper looks like this,

client.post("http://example.com")
.body("boo=foo")
//.headers(headers.clone())
.send();

You can see .body() method which is taking a &str. This won’t help me as I’d need to fill all the data in one single go and wait (blocked) till request completes. I don’t want that, so I started looking around to see how I can send the body in chunks.

I couldn’t find much resource online (didn’t spend too long) on how to send chunked data but then I stopped googling around and started reading code. The body method takes a generic argument body, which is bounded to use Into<Body<’a>> trait as shown below.

pub fn body<B: Into<Body<'a>>>(mut self, body: B) -> RequestBuilder<'a>

The Into trait is a convenience wrapper to convert into Body. The interesting part is really the enum Body.

pub enum Body<'a> {
/// A Reader does not necessarily know it's size, so it is chunked.
ChunkedBody(&'a mut (Read + 'a)),
/// For Readers that can know their size, like a `File`.
SizedBody(&'a mut (Read + 'a), u64),
/// A String has a size, and uses Content-Length.
BufBody(&'a [u8] , usize),
}

You can see in the declaration above the variants for type Body that you can pass to .body() function and ChunkedBody is one of them. To construct it, it requires a type that implements Read trait I started looking into it’s source code on how to implement it. So the Read trait’s definition is pretty simple,

fn read(&mut self, buf: &mut [u8]) -> Result<usize>;

It accepts a mutable buffer which gets filled in by our implementation and set the right usize as response. One thing to note was Result type is not from std::result::Result instead it’s from std::io::Result which is a typedef to the former to avoid writing std::io::Error whenever writing Result type within std::io::*.

Below is the whole code which implements Read trait and I was surprised it worked straight away! When timeout hasn’t been exceeded, it fills buffer and sends a Ok(usize) to indicate how much data has been written to buffer. It sends Ok(0) to indicate that it’s finished writing all the data(total_data_size) which is held in UploadData struct. It is set when UploadData is initialized. And it returns an Err when call site requests for more data to be filled into buffer but the time has run out.

This gives me a handle on tracking timeouts and also current_size tells me how much has been POST’ed which is used in calculations later.

I’m sure the code above could further be optimised and be more idiomatic. As I’m still working on it any comments to improve the code would be highly appreciated. Especially filling up buffer by iterating through it seems not great.

However the point of writing this up is to explain my thought process in following the Rust’s type system to solve a problem that I didn’t really have a full grasp of, especially using a new language and a library. The type system feels so simple to reason about. I wasn’t a big fan of subtyping and Rust is so much nicer to work with, without it.

And the compiler works like a charm and helps me a lot. I’ve had one crash so far but that’s because of me using unwrap() in the wrong context and it’s very easy to spot (no chasing rabbit holes as one does with segfaults). It seems like it follows Haskell’s (ghc) ‘if it compiles it will work’ sort of motto.

If you have any comments or suggestions to help the code above or to correct me in my understanding of any of the concepts discussed in this article please drop a line. I’ve more things to learn, so far it’s been really rewarding to work in Rust.