AI Case Study: Content Generation and Translation in Less Common Languages with noÁr
By: Han Wang, with Kate Gage, Oluwakemi Oso of Cooperative Impact Lab, and Dominik Kubik of noÁr
In the fall and winter of 2023/2024, Cooperative Impact Lab worked with a cohort of 13 organizations to support experimentation with AI tools ahead of and after the 2023 election. This post is part of a series of AI Case Studies documenting that work, highlighting lessons, best practices, and recommendations for organizations — especially those that organize and campaign — as they consider incorporating AI into their work.
In this case study, we examine using ChatGPT to create and translate social media posts for a Hungarian social movement organization. While this post focuses on content generation and translation both from and into Hungarian, we believe the lessons here are transferrable to generation and translation in other languages. This post focuses on the use of GPT4 because it is (currently) available in Hungary, unlike Claude at the time of this project. Alongside general LLMs, there are also bespoke AI translation tools that may work better with your use case.
Background
noÁr, a Hungarian social movement organization, used the AI chatbot ChatGPT to generate social media posts for a call to action in Hungarian.
This case study will outline the process noÁr took, highlighting key learnings that can inform anyone trying to use
Goal: Use ChatGPT to generate social media posts in Hungarian that encourage a call to action
Results: Mixed.
First Approach: Use ChatGPT 4 to prompt directly in Hungarian.
noÁr’s first approach was to prompt ChatGPT 4 in Hungarian to write a social media post about Gábor Iványi (the founder pastor of Hungarian Evangelical Fellowship, who is a target of the Hungarian Government), and then wrote a second prompt to ask ChatGPT (also in Hungarian) to generate 3–4 social media posts with calls to sign the petition. Some of the output was unintelligible — in one example noÁr told us the highlighted text below is “not a Hungarian sentence, just random words.”
Second approach: Add additional context to the prompt
Iterating on the first approach, noÁr uploaded text from their website to provide ChatGPT with additional context about the political situation. They also adjusted the prompt to add more detail about the people involved and then asked for social media posts with calls to action. This approach generated better posts, but ChatGPT made grammar mistakes. For example, it used the wrong suffix for the highlighted word:
Third approach: Ask ChatGPT to translate prompts written in Hungarian into English and then translate English results back into Hungarian
Based on the initial results, we hypothesized that ChatGPT might not be well-trained in less common languages such as Hungarian (compared to say, English or Spanish) and this relative paucity of data causes ChatGPT to produce suboptimal results. While ChatGPT is supported in Hungary, OpenAI has not published an official list of supported languages. This is understandable, based on the way that ChatGPT is trained on data from the internet, which is largely in English.
Working with this hypothesis, we explored methods to get ChatGPT to do the heavy lifting in English, which in theory would return better results. The strategy we landed on was:
- Write the prompt in Hungarian
- Ask ChatGPT to translate that prompt into English
- Run the prompt in English, and
- Translate the results back into Hungarian
noÁr used the prompt below to power this workflow:
Response from ChatGPT:
noÁr was happier with the results from this iteration. ChatGPT did continue to make mistakes, such as translating the metaphorical phrase “his journey” in English to a more literal meaning of “traveling” in Hungarian.
Conclusion
After these experiments, noÁr decided that the results were usable for their campaign with some editing and that it is a useful workflow going forward. This finding is in line with the experience of other cohort members who use AI as an assistant to augment their capacity, rather than a standalone content creation machine.
In our view, the double translation workflow is a useful template for how to use AI systems in various languages. Other CIL cohort members reported that developing prompts in their native languages and then having AI translate the prompt into English was an effective alternative to trying to develop prompts in a second or less familiar language.
This post will be updated as we and our partners experiment with different models, tools, and languages.
Thank you to our partners at noÁr, Trestle Collaborative, AI Impact Lab, Shamash Global, and Zinc Labs for their work and support on this project.
To learn more, or get regular updates from CIL on our work, please contact us at https://www.cooperativeimpactlab.org/contact
