Go MTOM Attachments
One of the common problems we face as an InsureTech start-up is how to wed modern technologies and tooling appropriate for a fast-moving start-up with outdated insurance industry or enterprise technologies. The insurance industry is complex, and I’ve found one of the most challenging and rewarding things about my job as an InsureTech engineer is to find the right way to abstract away industry complexity.
In this article, I’m going to outline what MTOM (Message Transmission Optimization Mechanism) is and how we built a Go client to send MTOM messages in order to integrate with an insurance vendor.
The Use Case
We recently partnered with Brighthouse Financial to offer a life insurance product, Brighthouse SimplySelect, which doesn’t require a medical exam and can provide an underwriting decision in under 24 hours (a process that can normally take upwards of a month). As a part of this integration, we needed to submit finalized applications to a 3rd party for processing, using an insurance industry standard XML format called ACORD. The ACORD format seeks to standardize industry concepts and entities into a common format to help facilitate integrations between insurance carriers, vendors and brokers like Policygenius.
One thing we’ve learned about ACORD is that while it’s a standard, there’s still a lot left to interpretation. Each integration using ACORD will require its own variations and extensions on top of the ACORD standard (understandably so — you try creating a totally rigid API schema for a $1.2 trillion industry).
One such aspect of our Brighthouse SimplySelect integration that required custom functionality was how signed forms were included in the application submission. In order for applications to be processed successfully by a downstream 3rd party, signed forms needed to be included as MTOM attachments that were then referenced within the body of the XML request.
What are MTOM attachments, exactly?
After initially hearing about this requirement, “MTOM” was a meaningless series of letters to me, like “YOLO” or “TTYL”. Lucky for me, the internet exists, and it informed me that it stands for Message Transmission Optimization Mechanism. Cool, I guess, but still pretty meaningless. After some more digging, I found a more useful definition:
MTOM, or Message Transmission Optimization Mechanism, is a SOAP protocol that utilizes the XOP “include” standard.
What this means in practice is that MTOM is a mechanism that embeds binary data into the body of a request and links to it from within the SOAP envelope. The benefit with utilizing MTOM is that it provides better performance and over-the-wire efficiency compared to the common alternative of base64 encoding a file and embedding it directly into the SOAP envelope.
The server that’s receiving an MTOM request needs to know that it should read each piece of the request body as separate attachments, with their own file type. The MTOM format does this by dividing the request body with MIME boundaries that describe the data that follows it.
For example, if an XML request needs a PDF and JPEG image attached via MTOM, the first MIME boundary would include XML, then a MIME boundary would say “hey server, read the gobbledygook you’re about to see as a PDF”, then a third MIME boundary that indicates the binary data is a JPEG image.
Each of these MIME boundaries should have a unique Content-Id that you can reference in your XML. The benefit of that is that you can use the structure that XML provides to provide context to the attachment.
Imagine if an XML document described an application for a home insurance policy and an auto insurance policy — with MTOM, you could associate attachments to either the home or auto policy application by linking to that attachment in the relevant parts of XML. This also allows you to add metadata to the request relating to a given attachment, such as form codes or signature dates.
A common alternative to this MTOM method is to just base64 encode file data and embed directly into the XML document, though this can put stress on XML parsing/schema validation functionality in the downstream server.
In the case of our integration with Brighthouse, we needed to send a bunch of signed documents with a given application. For each document, we needed to include form metadata like form requirement codes in order for the downstream services to validate the application submission was “In Good Order” and met all the regulatory requirements for a life insurance application.
Implementing an MTOM Client in Golang
At Policygenius, we like to take a domain-driven approach to designing systems, which led to building a new Go service that would handle the “Accelerated Underwriting” process.
Not being able to find an off-the-shelf Go package that would fit our specific needs for building MTOM requests, we decided to build our own, influenced by gowsdl. I’m not going to outline code for the entire SOAP client, since there’s a thousand ways to skin a cat — instead, I’m just going to focus on the MTOM-specific aspects of our implementation.
So what do we know we need this MTOM client to do so far?
- We probably need to add some headers to the request to let the downstream server know this is an MTOM request, so it can handle it accordingly.
- We need to be able to generate a multipart MIME request body.
- We need to generate a content ID that’s both marshaled into XML and appended along with an attachment.
- We need to dump some binary data in said MIME part.
Configuring the Request Headers
We need to let the downstream server know our intentions with this request, since the multipart-ness of it would look pretty funny. Kinda like if you forget you are holding a glass of water after holding a soda, take a sip and are like WTF IS THIS?
There are different flavors depending on your implementation, but you’ll need a few ingredients to your mah’s favorite MTOM Content-Type header meatloaf:
startContentID := "some-id"
boundary := "Hey Im a new section now"
buffer := new(bytes.Buffer)
url := "https://www.somehost.com/some-endpoint"// buffer is empty for now- this is where we'll do our marshaling
req, err := http.NewRequest("POST", url, buffer)contentTypeHeader := fmt.Sprintf(`multipart/related; start-info="text/xml"; type="application/xop+xml"; start="%s"; boundary="%s"`, startContentID, boundary)req.Header.Add("Content-Type", contentTypeHeader)
In the code above, we’re just creating a new request and passing it an empty buffer — we’ll be marshaling the request body in that buffer in a later section. Finally, we’re setting the request’s ContentType with a few values:
multipart/relatedtells the server this body will be broken up into a few different sections.start-info="text/xml"is instructing the downstream server that the first section of data will be XML. This’ll be where the SOAP envelope goes.type="application/xop+xmlthis is basically just informing the server that there’s going to be some XML and some attachments in the body.start="some-id”when defining the boundary for that first SOAP envelope section, we’ll need to specify aContent-Idthat correlates to this name.boundaryis used to indicate to the server how the request is going to be designating boundaries between sections of the request. More on this in a bit.
Marshaling the Attachment
Since we need to both 1) add the attachment binary(s) in a MIME part to the request body, and 2) reference that attachment in the XML, it makes sense to leverage Go’s XML marshaling interface. We can build out a struct that, when marshaled, can generate the correct Content-Id references in our XML and create the MIME boundaries.
First, we need to build a struct that is responsible for:
- Temporarily storing the attachment data
- Generating a content ID
- Marshaling into a XML node that links to said content ID
type Attachment struct {
content *[]byte
contentType string
partID string
name string
}func NewAttachment(v []byte) *Attachment {
return &Attachment{&v, "application/octet-stream", "", ""}
}
Next, we need to add some code to handle this custom marshaling behavior. Specifically, where ever we have an Attachment struct in our SOAP envelope, we want to translate that into an <Include> XML element that references the ID of one of the multipart sections (we’ll add these in the next step). We can add a MarshalXML method to our Attachment struct that implements go’s xml.Marshaler interface:
type AttRef struct {
Include struct {
Href string `xml:"href,attr"`
} `xml:"http://www.w3.org/2004/08/xop/include Include"`
}func (a *Attachment) MarshalXML(encoder *xml.Encoder, start xml.StartElement) error {
a.partID = fmt.Sprintf("%d", rand.Int())
attachmentRef := AttRef{}
attachmentRef.Include.Href = fmt.Sprintf("cid:%s", a.partID)
return encoder.EncodeElement(attachmentRef, start)
}
Now, when we marshal a struct that represents our SOAP envelope, our new MarshalXML method will be called when it runs into one of our Attachments as it navigates the nodes in that SOAP envelope struct.
Creating the Multipart Request Body
Now, we need to build a custom encoder that generates proper MIME boundaries, marshals our SOAP envelope, and scoops up any MTOMAttachment elements and throws their sweet, sweet binary data into separate MIME parts. First things first, we need to update our code that builds the request to encode an envelope struct.
startContentID := "some-id"
boundary := "Hey Im a new section now"
buffer := new(bytes.Buffer)
url := "https://www.somehost.com/some-endpoint"encoder = newEncoder(buffer, boundary, startContentID)
encoder.Encode(envelope)
req, err := http.NewRequest("POST", url, buffer)// ...
For brevity’s sake, I’m going to skip over defining envelope. All you need to know is it should be a go struct that is structured like a SOAP envelope and includes at least one of our Attachment structs. You can find an example of the basics of XML marshaling in Go here.
Finally, we need to build the encoder that’s being called above. First, we’ll need to set up the main interface for the encoding functionality:
type encoder struct {
writer *multipart.Writer
boundary string
startContentID string
}func newEncoder(w io.Writer, b, id string) *encoder {
return &encoder{
writer: multipart.NewWriter(w),
boundary: b,
startContentID: id
}
}func (e *encoder) Encode(envelope interface{}) error {
return nil
}
Next, we’ll need to add to our Encode method so that it takes the struct representing our SOAP Envelope, marshal it into XML, and write it to the first part in our multipart request:
func (e *encoder) Encode(envelope interface{}) error {
partHeader := e.getXMLPartHeader()
partWriter, err := e.writer.CreatePart(partHeader)
if err != nil {
return err
} xmlEncoder := xml.NewEncoder(partWriter)
if err := xmlEncoder.Encode(envelope); err != nil {
return err
} return nil
}func (e *encoder) getXMLPartHeader() textproto.MIMEHeader {
header := make(textproto.MIMEHeader)
header.Set("Content-Type", "application/xop+xml; type=\"text/xml\" charset=UTF-8;")
header.Set("Content-Transfer-Encoding", "8bit")
header.Set("Content-ID", e.startContentID) return header
}
The Encode method now creates a new section in the multipart request and writes the marshaled XML to it. Because we previously defined a MarshalXML method on Attachment, the XML encoder here will marshal this field as an <Include> XML node that links to the attachment binary in a subsequent part in the multipart request. The only thing we have left to do now is add data for those attachments to their own request parts.
func (e *encoder) writeAttachmentParts(pw io.Writer, d interface{}) error {
attachments := make([]reflect.Value, 0)
detectAttachments(d, attachments)
}func detectAttachments(data interface{}, attachments *[]reflect.Value) {
val := reflect.Indirect(reflect.ValueOf(data) switch val.Kind() {
case reflect.Array, reflect.Slice:
for i := 0; i < val.Len(); i++ {
el := val.Index(i)
if !el.CanInterface() {
continue
}
if _, ok := el.Interface().(*Attachment); ok {
*attachments = append(*attachments, el)
} else {
detectAttachments(el.Interface(), attachments)
}
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
if !v.Field(i).CanInterface() {
continue
}
f := v.Field(i)
if _, ok := f.Interface().(*Attachment); ok {
*attachments = append(*attachments, f)
} else {
detectAttachments(el.Interface(), attachments)
}
}
}
}
We first need to navigate our envelope in order to collect references to all of the Attachment nodes in it. We need these so we can match up the Content-Id attribute in the new request part with the cid reference in the marshaled XML’s corresponding <Include> node. We need to use reflection in order to recursively navigate the struct and find all the attachments. The switch statement is here so that we can also navigate arrays in our envelope structure.
Finally, we want to iterate through these attachments we found in the envelope and add them as new request parts.
func (e *encoder) writeAttachmentParts(pw io.Writer, d interface{}) error {
attachments := make([]reflect.Value, 0)
detectAttachments(d, attachments) for _, attachment := range attachments {
a := attachment.Interface().(*Attachment)
header := make(textproto.MIMEHeader)
if a.contentType == "" {
a.contentType = "application/octet-stream"
}
contentType := fmt.Sprintf("%s; name=%s", a.contentType, a.name)
header.Set("Content-Type", contentType)
header.Set("Content-ID", fmt.Sprintf("<%s>", a.partID))
header.Set("Content-Transfer-Encoding", "binary")
header.Set("Content-Disposition", fmt.Sprintf("attachment; filename=%s", a.name))
if pw, err = e.writer.CreatePart(header); err != nil {
return err
}
pw.Write(*a.content)
}
}
Now all we need to do is add the writeAttachmentParts method to our Encoder:
func (e *encoder) Encode(envelope interface{}) error {
partHeader := e.getXMLPartHeader()
partWriter, err := e.writer.CreatePart(partHeader)
if err != nil {
return err
}xmlEncoder := xml.NewEncoder(partWriter)
if err := xmlEncoder.Encode(envelope); err != nil {
return err
} e.writeAttachmentParts(partWriter, envelope) return nil
}
Now that our Encoder is properly marshaling our binary data to <Include> XML nodes that reference subsequent attachments, we should be all set to build and send MTOM attachments!
Building integrations in the InsureTech space offers a lot of interesting problems to solve. If you’re interested in helping us solve problems like this, take a gander at our openings and let us know if anything sounds like a good fit!
