How to get a multipart file size in Golang

OK, so your asking yourself, why… that’s easy with os.File you can just use the Stat() method and get the os.FileInfo and inside the os.FileInfo struct there is a Size() method and that will return an int64.

file, err := os.Open("path/to/file")
if err != nil {// Do Something}
fInfo, err := file.Stat()
if err != nil {// Do Something}
fmt.Println(fInfo.Size())

That’s it! Done! post over…. right? Well No…

Getting the size from a file on your file system is trivial, but many of us are getting a file over an HTTP connection from an HTML form. To add to this complexity, what if we are not persistently saving the file to the file system on our service, but pushing it to a 3rd party storage such as AWS S3. Now things are much more difficult.

To start off we would use "mime/multipart" package and receive a multipart.FileHeader to receive the file data.

type FileHeader struct {
        Filename string
        Header   textproto.MIMEHeader
}

REF: https://golang.org/pkg/mime/multipart/#FileHeader

As you see you get a FileName string and a Header textproto.MIMEHeader (which is a map[string][]string). In my experience the FileHeader.Header map may not contain a size. If it had a size it would have come from the client, and you shouldn’t trust what comes from the client. Also using the multipart.FileHeader means your service may not have completely received the file, especially if it’s a large file (i.e. >1GiB files).

When receiving multipart files, we receive it in chunks. This means when we want to receive a HTTP form we have to define how big the chunk size of the data we will receive.

http.Request.ParseMultipartForm(int64) // Chunk size in bytes

REF: https://golang.org/pkg/net/http/#Request.ParseMultipartForm

Typically with file data we will set this to 10MB. Now this is great, but now we 2 issues. What is the file size exceeds 10MB, and what if the file size is smaller than or equal to 10MB? For this we need to look the godocs of mime/multipart package, and see if we can get something to give us some sort of info on the file. We know we have a multipart.FileHeader and we see that it does actually have an FileHeader.Open() (https://golang.org/pkg/mime/multipart/#FileHeader.Open) method, and t

That returns a multipart.File (https://golang.org/pkg/mime/multipart/#File)

type File interface {
        io.Reader
        io.ReaderAt
        io.Seeker
        io.Closer
}

But a new issue arises, looking at the source code of the FileHeader.Open(), we see there are 2 possible outcomes.

func (fh *FileHeader) Open() (File, error) {
    if b := fh.content; b != nil {
	r := io.NewSectionReader(bytes.NewReader(b), 0, int64(len(b)))
	return sectionReadCloser{r}, nil
    }
    return os.Open(fh.tmpfile)
}

REF: https://golang.org/src/mime/multipart/formdata.go?s=592:649#L137

First we need to see the multipart.FileHeader with is exported, and un-exported fields.

type FileHeader struct {
    Filename string
    Header   textproto.MIMEHeader
  	
    content []byte
    tmpfile string
}

REF: https://golang.org/src/mime/multipart/formdata.go?s=592:649#L128

Now we see that there are 2 additional fields that are un-exported, one of them being FileHeader.content that is a []byte. That seems to be the streamed chunks total, and we could get the size from that I’m sure, but we can’t access that. Going back to the FileHeader.Open() method we see we can either get an un-exported type multipart.sectionReadCloser, which is un-exported, so we can’t access it.

type sectionReadCloser struct {
    *io.SectionReader
}

REF: https://golang.org/src/mime/multipart/formdata.go?s=3074:3116#L157

As you can see it is just an imbedded io.SectionReade, but we do get the multipart.File back regardless. Or we get an os.File that at the beginning of this post we can get a size from the os.FileInfo.Size() method.

So looking at the source of the FileHeader.Open() method we see that is the file size is larger than the defined chunks then it will return the multipart.File as the un-exported multipart.sectionReadCloser, or as an os.File. While we do get always get back a multipart.File back we actually can play around a bit with the os.File as well.

To get the size from the os.File we can cast the our outputted multipart.File, but this will only work if the file size is smaller than the defined chunks. Well to make sure this works we will need to make a conditional to see if it is cast-able to an os.File.

In comes type switching: https://golang.org/doc/effective_go.html#type_switch

This allows us to check if a type can be cast to another type.

mfile, _ := fh.Open() //fh *multipart.FileHeader
switch t := mfile.(type) {
    case *os.File:
        fi, _ := t.Stat()
        fmt.Println(fi.Size())
    default:
        // Do Something
}

Now we’ve checked if it can be an os.File then we get the size from the FileInfo.Size() method. But if it is not cast-able what do we do?

Looking at the what the multipart.File interface implements, we can seek the io.Seeker type. All that does is just implement the Seek() method. The params needed are an offset int64 (this is at what byte you want to start counting), the whence int (this is the byte where you want to end up), and outputs the length and an int64 (and an error). So here is what the default in the switch will look like:

...
    default:
        var sr int64
        sr, _ := mFile.Seek(0,0)
        fmt.Println(sr)
...

Ok so you see we don’t use the t var created in the beginning of the type switch, this is because we need the actual pointer. Second the offset param is 0 because we want to start at the beginning, and the whence is also 0 because the Seek() method goes through the bytes then goes back to start. THATS IT!

Side Note: As you can see here we had to do quite a bit of investigating. That is a huge benefit of Go. It is so well documented, and can create really interactive docs automatically. Go also makes it easy to go through the source code and see how these methods output. That is in large part the face that all of the standard packages are written in go. Also the fact the the developers meticulously comment their code to make it easily readable. Finally there is a great webpage (https://golang.org/doc/effective_go.html), this clearly defines Go’s best practices, and should be carefully studied by any Go developer for clean consistent code.