Translating Codecademy
Lessons learned bringing code to the world
Translating Codecademy
At some point most companies make an effort to “go international”, and a key part of this effort is translating site content into local languages. Codecademy never had to make this choice — one day after launch the site was being used in over 170 countries. Soon after, people from around the world wrote to the company graciously offering to translate coding lessons into their local language. Over the past few years, crowd translation has risen in prominence as a low-cost but accurate way to translate content. The reality is slightly more complicated.
Codecademy Crowd Translation v. 1.0
Crowd translation works well when the content can be divided into small independent chunks and there is a high degree of feedback for each chunk of translated content. The other option is to use professional translation services, which usually are more expensive.
Codecademy’s content receives a high degree of feedback but had many highly dependent phrases, with consistency required between lesson instructions, hints, default code, and code to check user submissions to pass the user onto the next lesson (“submission correctness test”). Despite this hurdle, people volunteered en masse, and did a phenomenal job translating the site into dozens of languages.
As is expected with anything done for the first time, there were problems with this translation pass:
- Translations of identical phrases were inconsistent
- Individual translation projects usually did not reach 100% translation completion
- Using multiple translators sometimes meant a coding exercise was impossible to pass because the instructional text did not match the submissions correctness test
Also, Codecademy was operating under extreme resource constraints — when first localizing content Facebook had a staff of 500, Dropbox had a staff of 50, and at the time Codecademy had a staff of 20.
Codecademy Crowd Translation v. 2.0
After our first efforts at crowd translation, we decided to use a hybrid translation approach for the next iteration instead of pure crowd translation. Our hybrid approach involved using translation partners for the language we wanted to translate (e.g., Lemann Foundation for Portuguese, and Bibliothèques Sans Frontières for French) to coordinate multiple translators, and ensure translation quality and completion. The strength of the translation partner was part of additional criteria we used to prioritize which languages to translate. We also started implementing translation tools to make the job easier for everyone.
Translation Criteria
There were three items we considered when deciding which languages to translate:
- Users on Codecademy from a specific country speaking a specific language — this is actually harder that it may sound. For example, users from India are well represented on Codecademy, but it can be difficult to tell whether those users speak Hindi, Bengali, Telugu, etc. The other reason we focused on individual countries is partnerships and promotional campaigns, and pro-code literacy legislation happen on the country/city level.
- Total country population and language speakers globally — we wanted to pick languages with huge growth potential regardless of geography. For example, Argentina has a population of ~40m people, but Spanish is spoken by ~400m around the world.
- Strength of local implementation partner — we looked for local partners who had worked on translation projects with high quality results.
Translation Tools
LocaleApp
There are numerous translation platforms that enable crowd translation, and we chose LocaleApp. After the content strings were extracted, LocaleApp allowed multiple translators to work on the strings at the same time. However, it was difficult to see the strings in context. The only way for a translator to view the quality of the translation was for us to upload the translated content, and then to review the strings on the site.
Get Nikhil Abraham’s stories in your inbox
Join Medium for free to get updates from this writer.
Exercise pass rate
Exercise pass rate was our secret translation weapon. Most sites that use crowd translation need users to review content and upvote/downvote the translation. Because users are constantly using our site to complete exercises, we monitored pass rate before and after translation to see if there were any big differences. If an individual exercise suddenly had a 10% pass rate when previously the pass rate was 70%, a translation error was most likely the problem, and the exercise was flagged for review.
Moderator community
The glue holding all these mechanisms together was and continues to be our strong moderator community, the group of experienced super users on our site who help new users. Having community members on the front lines monitoring user feedback and going through new exercises is extremely helpful for identifying which problems are most severe.
Problems with Crowd Translation v. 2.0
Our second pass at translation definitely went better than the first, but there were still problems.
Handling conflicting versions of translations
In our current process, we always assumed that translators would finish translating all content before upload, and translations would only improve over time. In almost every case, translators wanted to see how the translated version would appear on the site before completion, and in a few cases we had overwritten correct translations with incorrect ones and had to revert to an earlier version. We had not implemented a clean versioning system, so finding backups was time consuming and involved lots of digging.
Not translating all UI/UX strings
Content extraction and then display of translated strings are key steps when doing any translation project. After translation, we noticed that some site UI/UX buttons and modals were not translated properly. Most of our content is pulled from YAML files using the Rails i18n API, but some of our older site UI/UX navigation elements were hard coded into HTML templates. This meant that when strings were initially extracted for translation these hard coded strings were not included.
Serving up the wrong locale
After everything was uploaded to the site we started hearing comments from some users (included Zach who was in Brazil at the time) that they were not seeing the fully translated version of the site. We had mapped out how to set locales, but not every way a user thinks about locale. Localized versions were shown, in order of importance, when a user
- visited a localized TLD (e.g., codecademy.fr),
- requested a localized version of Codecademy using our language selector, and/or
- had a localization preference set in the browser
We thought TLD visits and users selecting a language would be the bulk of international traffic, but the data proved the exact opposite. As the red line shows, most users retrieved a localized version of the site based on the language preference set in the browser.
As separate issue, our content architecture allowed users to stumble upon Portuguese content even when the locale was set as English. Additionally, in some cases we were serving the fallback locale (pt) which had not been translated, instead of more specific locales (pt-BR).
Hopefully, you are starting to get a sense that localization is not as easy as translate it and forget it.
Codecademy Crowd Translation v. 3.0 and beyond
There were many lessons we learned while localizing the site. To make the next translation iteration even smoother we have built a content versioning service, and monitoring scripts to watch for big spikes or drops in locales served or exercises completed in a different languages. We are also thinking about additional tools we can build to help our translators, and these are questions we are tossing about:
- How can we show translation previews before all the content is translated?
- How can we help translators who have little coding experience translate code in a consistent manner?
- How can we point out exercises that would be impossible to pass because of a translation error before having thousands of users fail the exercise?
Our process is still far from perfect — reach out and let us know if you want to chat about better solutions you have for some of our challenges!
