Confidently Implementing Machine Translation for Commerce
In today’s cloud-based world, many companies are changing their web content daily or even multiple times per day. The rapidity of content change is stretching marketing and go-to-market teams even in the source language, and often breaks down when publishing regionally.
Confounding the issue, online consumers have constantly evolving needs and expect relevant, even personalized content. When they don’t find it in one place, they are willing to go elsewhere. Brand loyalty takes a back seat to the want-it-now mentality. For this reason, the desire of sales makers to harness the global reach of the Internet is at odds with the tried and true human translation workflow systems most companies have historically used.
Even so, marketing and localization teams are hesitant to trust machine translation (MT) to deliver expected quality, even though they understand the time advantage. BLEU scores and its ilk are meaningless when seeking to gain confidence in automation — rather, forward-looking localization leaders have to speak the language of eCommerce.
The true potential of MT can be demonstrated using web analytics and A/B testing, which looks at two statistical variants in a controlled experiment. You need to present consideration and conversion metrics in terms that marketers and sales makers understand. A/B testing, or test and target, gives marketers website optimization capabilities to continually focus their online content and offers more relevancy to their customers, yielding greater conversion.
For our purposes, designing a test setup is fairly simple. First, you decide which language you want to test and have the website content translated both through the traditional human translation process and by MT — likely post-edited MT. Second, you use test and target to send 50% of the visitors to the human translation content and the other 50% of the visitors to the post-edited MT content. At the same time, you run a website survey for the visitors who get to see machine-translated content, and you collect their feedback. Finally, you analyze the site traffic to each set, looking at click-through, click-away, and conversion for each.
Now that you have meaningful data, it’s time to build the business case. We already know the post-edited MT process will be cheaper — how much so depends on a realistic judgment of how good the quality must be — and we know that the process should be quicker, or at least it will be after post-editors get through the learning curve. So now we have to weigh those advantages against the test outcomes.
From testing to action
We found that the key metrics between the test sets were essentially equal, meaning both types of translation resulted in roughly the same customer behavior. Your results may vary, but the more tech-savvy your customers, the more likely it is that they will prefer function over form. Even if post-edited MT is at a slight disadvantage, you have to consider that if the speed and cost advantages allow you to localize more or get to market quicker, this will increase your revenue even with a slightly lower conversion rate. Additionally, you can always go back and improve the translation for higher volume products.
After proving the benefits to ourselves, we threw caution to the wind and moved French and German fully to post-edited MT. We stood back and waited for the world to end, but after two quarters of holding our breath with not a single escalation or complaint, we set an aggressive strategy and moved website product content to post-edited MT for all 27 languages (now 28) within a years’ time.
It’s a great success story, but it’s not as simple as it sounds, and there was a lot of groundwork to be laid first. Let’s look at some of the steps and challenges to making this successful on an enterprise scale.
First, yes, there are trade-offs. Now is the point when everyone will say “see, I knew it!” MT technology has improved greatly over the past decade, but it still falls short of human translation quality, and some languages are just downright difficult for automated solutions. However, we weren’t forced to trade quality. We traded savings, accepting a very small post-editing discount as the engines ramped up and the translators made the transition. A couple of years into the program we are averaging about a 20% discount on new words, and while that may be small against what people expect to get in MT discounts, 20% of 60% (the percentage of new words we’d be translating from scratch) of our multi-million annual word volumes still adds up to a significant cost savings.
Additionally, we have seen our average turnaround time drop by more than 50%. That improved speed-to-publish is immeasurable in terms of revenue potential, but certainly garners a great deal of goodwill internally for a process that inevitably gets pegged as a choke point for content release. It’s unlikely we will gain much more in the way of cost improvements — it is eCommerce, and while function may be more important than form in certain industries, there is still an expectation of quality that can’t be ignored.
Nobody wants to buy something from a company with shoddy content. Consumers around the globe understand and accept that content is translated, but they still expect a certain level of dedication and investment in the local market. Sadly, we won’t be putting raw MT on the site anytime soon, at least not outside of support content.
Choosing an MT provider
One way to mitigate the potential quality trade-off is choosing a strong MT provider. All of our vendors leverage the same MT solution when performing post-editing. We made a number of stops along the way before finding the right fit. There is a large cadre of MT providers that do well with a few languages, and you may find yourself having to aggregate several solutions to meet all of your language needs. Many companies do this with internal linguists setting up and maintaining engines, but we believe that solution often leads to a disjointed approach. We took the route of having one of our language service providers (LSPs) source and test a number of technologies and finally settled on a single technology provider that is able to develop engines for all of our required languages. This greatly reduces management overhead on our part, and vastly simplifies tool integration.
Even with the right technology provider chosen, there are still a number of hurdles to traverse: how you integrate the tools; how you set up the translation memory (TM) sequence for optimal performance; and determining if you’re going to measure MT quality the same way as your human translation workflows, to name a few. Tool integration could make for an article or perhaps even an entire issue by itself, and is often dictated more by IT constraints and legacy tools to work around than it is by dollars or know-how. We were able to connect our MT tool through a simple application programming interface (API) to our translation management system (TMS), which is already integrated with key content management systems (CMSs). Content flows from the CMS into TMS, out for MT, back for post-editing and back into the CMS with no manual steps outside of the actual post-editing itself.
Integration costs were minimal, and oftentimes we could convince our partners to help shoulder the cost since more often than not, the API would benefit other clients. Even without help, it doesn’t take many man hours to justify an integration over manual workflow steps. On the plus side, once the integration is enabled, you can plug the MT process into workflows as needed, leaving off post-editing when quality expectations allow. It becomes just another workflow stage, like vendor authorization or applying TM. Because we demand a fairly high quality product from post-editing, we have the confidence to put the MT TM into the sequence, where more often than not, localization teams make the decision not to leverage anything resulting from an MT workflow.
Our TM sequence goes something like this: first the primary TM, then the secondary TM — the split between primary and secondary is more about managing file sizes and making TM cleaning easier than it is about a difference in the content in each — and lastly the MT TM, with no penalty. Of course, we only leverage the MT TM for the MT workflow. We owe it to ourselves to analyze what percentage of the leverage is coming from which TM, but that’s a task for later.
