In a world where language learning opportunities are limited to unpredictable stressful encounters and channels that fail to equip learners with real-world vocabulary, it is essential to understand that acquiring a language is rarely a linear process. With the use of formal language lessons and abstracted situations, current language learning practices were proven to lack context, enabling an endless loop of lost motivation. As a response, Retrio is an attempt for a more contextualised experience. This work in progress presents a digital scenario-based retrieval system for informal language learning. It is a personal library allowing people to keep track of their daily encounters and turn them into a language learning experience. This application gives the freedom for users to capture information in various mediums (text, audio, visuals…), to consequently organise them through personalized “scenarios”.

Introductionary video

User journey — a wander around the Hong Kong underground.

And that’s it, folks! It’s been a crazy, intense journey, and I’m sad it’s already over. Endless thank yous to my amazing, amazing project partners (Kate & Tatiana), my course tutors (John 🐙, Al, Mor and Greg) as well as the rest of our cohort. I couldn’t have done any of this without you all!

Over and out — see you on the next adventure!

Once we had received the photos, notes and recordings from our participants we got to work placing them into our designs. The information covered three situations:

• Taking the MTR (subway) in Hong Kong
• Getting lost and asking for directions in Seoul, South Korea
• Figuring out a weekly meal plan in Beirut, Lebanon

We were pleased to see that there was a wide variety of situations recorded, and that the data fit nicely into our designs.

Testing

We needed to be able to confirm that users can use and understand the prototype without an in-depth explanation from us. Thus, it was important to test our designs to ensure this.

For ease of testing (and lack of time) we decided to prototype only one of the three situations collected: the Hong Kong MTR. Our prototype, being its content is now situated in Hong Kong, was mostly in Cantonese. This narrowed our testing audience to those who are learning, or can speak Cantonese. We were able to find three people to test with: my brother (who gave me the data), my father (who also lives in Hong Kong, and is learning the language), and Sylvester (from our course, who is fluent in both Mandarin and Cantonese).

The test was successful, with every participant stating that they would use this app themselves. Two participants mentioned, however, that the organisation of information within the scenarios needed to be developed, as it currently has minimal information hierarchy.

Two participants noted that they felt a community aspect was missing, as right now it is a solitary experience and you cannot easily find out if what you are learning is incorrect. Due to time restraints, we were not able to prototype the collaborative aspect we discussed last week. Although this was a shame, we decided that this was the best plan to end with a more complete final outcome. However, we agreed that this feature would have added more depth to our project.

Finally, we noticed that during testing we needed to provide a fair amount of explanation for the participants to understand what scenarios are, and their functionality. This was frustrating to hear, however, we took this onboard and thought about how we could improve this. It could be because we had not prototyped an introduction/onboarding process to the app that would explain to a user what things are, and how to use them. Because we did not have sufficient time to fully design and implement this change, we decided to record an introductory video to the app to help explain key features. Although I acknowledge that it would have been better to implement changes that augment the design and flow of the app to be more intuitive, we simply did not have the time to do so.

Next steps

Because we were facing a looming deadline in a couple of days, we could not develop the community or explanatory aspects to the experience (especially because we also had to balance our work load with writing our essays!). Our next and final steps were to fill out gaps in the interactions on the prototype so that it would be ready for the video we wanted to film.

>>> For my next, and final post on the project, click here.

<<< For my previous post on the project, click here.

Last week we decided that our design would be a scenario-driven retrieval system for people who are learning a language informally. However, I was struggling with how we could make the most out of seeing visual context. Given that I begun the project with an interest in visually contextualising language learning, I wanted to explore this as much as I could with our remaining time.

Last week I started by making a scenario for meeting somebody new, but found this so hard to visualise — perhaps because the topic was too abstract. I made a new mind-map on the much more specific topic of coffee to help me ground my visualisations. I created categories for coffee preferences, ordering a coffee and talking about coffee, with sub-categories that get more specific, going into climate change, growing coffee and coffee notes & flavours.

However, it was still very difficult to bridge the gap between what I created, and an interactive prototype. We decided to start prototyping without this aspect figured out, as we needed to have something to present on our Thursday feedback session.

Feedback

We introduced our ideas from this week as above, presented as an app. This is not the only format we envisioned it to be made into — currently targetting touch screens as they’re more tactile. We were immediately critiqued for not including data from real people outside of our team. This is something we did not have time to do, but agreed that was extremely important, and would make our next priority.

A couple of our tutors commented that our designs reminded them of a word cloud. They mentioned that they would be really interested to see how much of a language they know displayed in this way — visualising the non-linear journey of learning languages.

Finally, they highlighted a problem with the personal aspect of our library: How are people going to understand how to be, or act in certain situations if they only have their personal experience to rely on?

We thought that this could be solved through collaborative use, giving users the chance to correct, or add information in the app. However we noted that this could be weaponised, or lead to misinformation. We thought that adding a “Downloadable situation packages” feature might be a good start as this could provide a foundation for essential learning upon which the user could build. Mor mentioned that we could strengthen this aspect by including automated word or phrase suggestions, based on what the user inputs into the library — similar to song suggestions on Spotify based on what you like.

Next steps

We decided to prototype this collaborative feature as we felt it would contribute to the functionality of our experience.

Now that we had tackled this aspect, we needed to fill the app with real data. Each of us planned to ask one person (who lives in a country that they don’t know the language of) to take pictures, audio recordings, and make notes while doing an action — which could be anything.

>>> For my next post on the project, click here.

<<< For my previous post on the project, click here.

Our next plan of action was to figure out how to organise the sections from my brainstorm last week.

I thought about how the information would look, and how it could be structured in a digital system. Seeing as elements within a conversation are all linked, I needed to carry on these connections in the design:

I liked the structure a grid provided as it also enables a complex navigation system, with layers and depth. However, I had still not planned how the design would work with actual pieces of information.

To help myself start thinking it through, I begun by planning how it might work with individual words. Nouns are easy to contextualise, so I was keen to complexify by exploring how words could be placed depending on their relation to nearby words e.g. a basic example would be the word lovely, which I placed next to cat. This was an attempt to see words in relation to connected concepts. When a word is selected, other, non-related ones would fade out of focus to highlight those that are connected.

However, this got me thinking about the importance of seeing in contrast as well. Surely it is just as important to see words in relation to what they are not, to help understand their meaning? I tried to loosely prototype this aspect, however decided to leave it until I had spoken to the group.

Feedback

We met with John for a mid-week tutorial and he criticised our reliance on vocabulary, saying that this use case would be unlikely unless the individual was learning language in a formal way e.g. at school with vocabulary homework — for which we were not planning. He noted that the strength lies in different types of content, and asked us to consider the different ways for representing the recall for those things e.g. a tense would have a different way of being recalled to a sentence structure etc. He added that our designs lacked criticality, and didn’t cover their potential. It has been over simplified — too reduced.

Our group discussed this after the call, and agreed with his criticisms. We focused our target audience on those that are learning language informally, and begun discussing ways we could design a contextually-driven way of organising content — something different to just flipping back pages in a book.

Next steps

We decided to make a new aim, to help focus our next steps:

To create a scenario driven retrieval system for people who are learning a language informally.

We wanted to make a product that would enable the learner to bring in different information of their own, and create their own scenarios. However, to do this we still needed a better understanding of how the information would be structured.

We each thought of our own scenarios, and thought of what might be said within them. We expanded our wireframes to consider trickier subject matter than individual words to properly explore our ideas. In trying to make the idea real, I found it difficult to incorporate images, and other forms of visual context — finding them feeling superficial, or uninformative.

This would be something I focus on next week, as this was an important aspect to the experience that I did not want to lose.

>>> For my next post on the project, click here.

In addition to other feedback last week, we were warned that we were beginning to make assumptions about language learners and their motivations to study. So, we decided to conduct some Directed Storytelling to help us uncover biases by asking specific questions about their processes.

Directed Storytelling

We managed to gather 11 participants, and asked them about their most recent language learning experience. Our goal was to understand their process, and what parts of their language learning journey they found the most and least valuable. We probed for underlying reasons behind their explanations by giving thoughtful prompts, to guide and frame their thought processes.

We were able to find that many of the participants noted the necessity of visual parts in learning, as it helped with association, emotional attachment, and context. Without, it was difficult to associate words with what they mean in their heads. There was also a lot of emphasis on needing a person to practise with — someone who can provide a comfortable environment where you can make mistakes and get corrected.

Finally, several participants noted that they missed the cultural aspect with their learning methods. Without culture present they felt isolated from the language which lowered their motivation.

We found these results insightful, however they also left us with many questions. Although an important step in the process, it had unfortunately not helped us narrow our focus as of yet, instead helping us inform and back up decision-making later down the line.

As advised by our tutors last week, we now needed to focus on the transfer between the physical room and the digital network — needing to think about how people go from having a conversation to seeing information appear — organised — on their phones.

We begun making a basic timeline of how the process would look start to finish, however, we quickly started to feel overwhelmed. We still did not know how, why or where these two people would be talking, and that it isn’t very likely that someone might go to a place with lots of visualisation technology just to have a conversation. In our Tuesday feedback session with John, he mentioned that the visuals might also become a distraction: if you’re trying to listen or talk to someone and pictures start suddenly appearing on the walls that might end up being more confusing than helpful.

We realised that it would perhaps be better to focus on the digital library part. Not wanting to stray too far away from our paper prototype last week, we decided to explore the social circumstances of a language a little further. This made us think about how people might survive in places where they don’t know the language, yet have to do language intensive things. It would be interesting to explore what happens in these situations, and figure out parts that could be saved to the library —somehow saving the context that exists within the situation, even when it no longer exists in the present moment.

Wanting to explore this functional aspect, we needed to prototype designs that perform well for specific needs. To do this, we needed to first define some of these specific needs.

To do this I chose an example conversation, “Offering someone tea or coffee”, and tried labelling all the things I could from it. I separated things within that situation out into sections: words (adjectives, nouns and verbs), social (timing), culture (politeness), technical (conditional and imperative tenses) and situational (location-based).

I presented my brainstorm to my group, who agreed that we should have done this earlier as it helped us develop our understanding of the topic and its design opportunities.

>>> For my next post on the project, click here.

We were very interested in exploring the collective, person-to-person aspects of language learning as this was a largely missing from the learning methods last week. Situations hold a lot of rich visual information that is difficult to capture just with visuals: gestures, accents, and audio. So we decided to conduct an in-person body-storming session amongst ourselves to explore the physical aspect of our ideas, trying to bridge the gap between the digital and physical. We were also finding it difficult at this point to focus our ideas for the project, so we hoped this would help us narrow down and start designing.

We had three body-storming sessions:

• One where I learnt French with Tatiana
• One where Kate learnt Arabic with Tatiana
• And one where I practiced Russian with Kate.

Each session lasted 15 minutes, in which time we tried to immerse ourselves in different conversational situations and speak as much as we could in the target language. We provided ourselves with prompt questions in case we got stuck, and couldn’t think of what to say.

We discovered that the conversation prompts were very necessary, as even though we knew each other well we grew shy and embarrassed when asked to say something to the other on the spot. We also found that we all took notes during the session, so that we could anchor down what was being spoken out loud. This also helped the native speaker in the session explain to the learner what things meant by drawing and circling certain things on paper.

Based on these sessions, we were keen to direct the projects focus on placing an emphasis on conversation, and enabling the communication of one’s intentions as a central theme to our FMP.

We starting planning a two-part experience:

• Part 1: A visually contextualised conversation — visualisations appear on surrounding walls as the participants speak.
• Part 2: A personal library — saving and organising all the information from part 1 onto the participants phone to refer to later.

We prototyped it to research the idea through design:

We brought the prototype into class for our weekly feedback session, where we explained our creation: This part of the experience would take place in a white room with a table in the centre, around which two participants would sit. As they speak the things they would say would appear on the wall — words, pictures, doodles they draw out on screens in front of them… These elements would accumulate, becoming reference points on screen to remember what was said, helping to scaffold the conversation.

The experience would be a live data visualisation, which would hopefully help in understanding what the other person was saying. We planned to include props, and other extralinguistic elements that would structure, constrain, and ultimately allow interpretation in normal communication in addition to the images on the walls.

Feedback

Our tutors liked that we were exploring physical ideas, and the weirdness and memorability that can come with it. However, we needed to consider fluidity of use — how can it be used in multiple locations, and is it convenient? We agreed with this, unsure about the inconvenience of the physical experience as this might increase the amount of effort involved in learning — counterproductive to our aims of increasing motivation.

They also warned that the two-part experience might be too much to plan. This is something I was also worried about, so we decided to discuss this as our next plan of action.

>>> For my next post on the project, click here.

After a break in the summer holidays my group met up online to discuss our thoughts on the project. Both Kate and Tatiana shared my concerns about the project’s breadth, so we made a research question:

“Design a way to most effectively apply context to vocabulary learning in foreign language learning”.

This helped us clarify whether or not we were focusing on teaching a language — clarifying that it is not our aim to do so. We wanted to focus on creating an additional visual component to the learning process.

Learning Systems Testing

We decided to test three different language learning systems that were highlighted by our questionnaire results: an App, Podcast and Textbook. We were interested to compare these three methods because of the different forms of context used in each: App uses text, pictures and audio, the Textbook uses text and visual, and the Podcast only audio.

We managed to gather seven participants who would use one of these methods to study French over the course of one week. They were asked to fill in a multiple choice diary every day even if they didn’t study that day, which covered these topics. This was the best way we could think of to get participants to record their thoughts, which also required the least-amount of effort.

Synthesis: "Don't synthesize anything I wouldn't synthesize!

After the week had ended, we each arranged a call with the participants to talk through how they felt throughout the week. We had discussed arranging a group workshop following the week with all the participants, to get them to do some card sorting together. However we were advised in a tutorial that we needed to move past the exploratory phase of our project, so we should instead focus on synthesising what we had already.

We each made a Behaviour map to represent the information put in the diaries.

Duolingo

Learn French by Podcast

15-Minute French Textbook

Synthesis

All the participants noted that there was a lack of meaningful imagery. The Duolingo group mentioned that they were useful to illustrate the meaning of nouns, but were generally hard to remember because of a similarity in style. The textbook participants noted that they would have preferred watching a TV show as it presents a more cultural environment, and is more conversational. Finally, the podcast group noted the lack of visuals made them struggle to associate what they were hearing with its meaning.

All the participants felt that there was a definite lack in personalisation, which was noteworthy. They found it cumbersome to look back through past material, and wished there would be a function that would do this automatically as it often ruined their flow.

Motivation was difficult to measure. As is clear from the behavioural mapping, this varied greatly, especially in the podcast and app. The podcast listeners noted that they felt a little more motivated to study because their learning was passive — they could put it on in the background. Two of the participants from the textbook and app group said that they had no prior interest in learning French so this lowered their motivation.

Reflection

In hindsight our test could have been developed further by asking more specific questions relating to each medium to find more meaningful differences between them. If I could go back I would also let the participants choose any language they wanted to learn, rather than make them focus on French. We chose French because we thought this would be a good way to test motivation, letting us see participants who wanted to learn the language in comparison to those that didn’t. However, I think it would be more important to test motivation in relation to something they initially wanted to do, and find out where/why it went wrong.

>>> For my next post on the project, click here.

Just before the summer break, everyone in the class did Crazy 8’s together to get us thinking.

Although I was not entirely convinced by the ideas I had drawn out, they got me thinking about possible ways forward.

A singular brainstorm

My group decided to each spend some time on their own during the summer to think about the project. Because I was a little worried about how wide our scope was, I decided to explore its breadth and pick out some key attributes. I organised the experience into three parts, so I could discuss it further with my team after the holidays.

Part 1: Organisation

I imagined the experience to begin once a set of vocabulary was uploaded. At this point I had not thought of why they would have this set, other than it would be vocabulary they wanted to learn. The next step would be the organisation of these words, which would be linked by association. At this point I did not know if these links would be made by the user, or by the system. The hope with this was to create a “web of understanding”, to which you could easily refer.

Part 2: Input + Exposure

To help understand each one, the user would now have to create a visual library around the words. This would be to help contextualise the words by relating them to the scenarios in which they could be used. For example, for the words “feet hurt”, the user could upload (in no particular order):

• dictionary definitions for each word
• example sentences: “I kicked the door and now my feet hurt”
• a video of a person experiencing pain in their foot
• a cartoon image drawn with a star to visually emphasise pain
• and finally a selection of movie scenes showing the words “feet hurt” in different scenarios (Playphrase.me. 2021).

Each one varied in its effectiveness, but combined they helped provide a fuller picture of what the words mean. However, at the moment this idea is not quick to make, and would have to have a degree of automation if adopted as a project direction.

Part 3: Use

Language is experiential — it’s meant to be used. I thought the final stage of the experience would be to put it into use. I had two ideas: the first (on the left) was through a messaging game, where you are talking to a character (in the image above I chose an alien robot) that you can’t understand — the character would speak in gibberish. The goal of the conversation would be to teach them the language you are learning by trying to explain it with your previous collections of visuals (from part 2) & webs of understandings (from part 1). This idea was quite ambitious, and not yet properly based in research. Although, I was excited at how unusual the learning experience would be, which could potentially help its memorability.

The second idea (on the right) was to learn with a group of people who are all learning the same language. It would take place online, where a story would be generated based on each players’ uploaded vocabulary. The players would then be given a starting point for the story, with a goal to reach a pre-determined end-point. They would then have to use the language they are learning to progress the narrative in turns, until they reached the end-point. The interaction would be through typing, however there would be visuals generated above where the players would type that show the story typed playing out. This idea was also quite ambitious — I had no idea how the story, or accompanying images would be generated. However, like the first idea, I was excited by it and decided to leave it to explore with the group when we next meet up.

>>> For my next post on the project, click here.

References

• Playphrase.me. 2021. PlayPhrase.me: Site for cinema archaeologists.. [online] Available at: <https://www.playphrase.me/#/search>.

Market research

I mentioned in my last post that we needed to really focus on grounding our idea in the people that would benefit from it. We decided that doing some market research would be a good first step.

We narrowed our search down to just mobile applications, as at this point our envisioned outcome was digital. We chose the apps based on popularity — as judged by the App Store — and split our categorisation into four: Gamified platforms, Flashcards and SRS, Social, Chatting & Listening, and finally Full language courses.

I then chose one from each category and analysed it further.

Duolingo

Duolingo is a language-learning web & mobile app that utilises gamification, such as gaining points and competing with friends to incentivise the learner. Although there is a competing system, there is a distinct lack of communication with real people. The main method of learning is by translation from one language to another, which is paired with some illustrations that lack in depth.

Quizlet

Quizlet is a web & mobile app that lets its users create customisable flashcards, where they can input words and their translations. It is not specifically a language learning platform, but a lot of its users learn languages with it. Similarly to Duolingo, you learn through direct translation, with the addition that you can upload your own images to help make the cards more memorable.

Chineasy

Chineasy was created with the purpose of teaching chinese characters to visual learners. This is based on the reasoning that the human brain is able to memorise information better if it is put into a visual context. The learner is taught characters through illustration, with the drawings correlating directly to the meaning. However, it doesn’t teach you how to use the words in sentences, or how to combine them with other characters.

HelloTalk

HelloTalk is a language practice app — its core function not being to teach you a language, but rather to connect you with native speakers. It encourages conversation as its main feature, with in-chat tools to directly translate what the other person is saying, without having to exit the app to go to google translate. However, it doesn’t have any features in place to save new things that you learn, meaning they may get forgotten.

Summary

The market research combined with the questionnaire data we found from last week resulted in us drawing a few conclusions. First, in all the apps reviewed, Chineasy was the only one to make rich visual context a focal point. Meaningful visual context was highlighted as important in the questionnaire, so it’s notable that not many focus on this aspect.

Second, most of the apps do not have a save feature to help you easily retrieve what you just learnt. This was mentioned last week, specifically to say that there is a “lack of freedom in study material organisation & customisation”.

Final comments

In hindsight we should have put more effort into analysing learning methods and tools that exist outside of the digital, or app, realm. I believe it would have given us more insight in what the actual gaps in the market are, and could have benefitted the uniqueness of our project.

>>> For my next post on the project, click here.

Target audience

At this point we had played around with certain directions for the project, but had not paid defining a target audience enough attention. We begun the project thinking we wanted to target those who travel abroad out of necessity but end up struggling to learn the language of the country they’re in (even after years of living there). However it was difficult to find relevant secondary research, and focusing on this group would mean the additional consideration of differences between adult and younger learners. We decided to stray away from this target, and try to focus more on the visual side of our project — target those who consider themselves visual learners.

This was still much too vague, and came with its own set of problems: for example, wouldn’t most people benefit from visual context, not just visual learners? And what defines a visual learner? These were questions we would consider and research going forward.

Questionnaire

To help us reduce assumptions in decision-making, I created a short questionnaire (synthesised above). I asked basic questions to get a better idea of popular language learning methods, what they would change about their process, whether their study method included pictures or videos, and what points of the process they found useful, or not. Finally, I asked them to rank in order of importance what (out of pictures, videos, example sentences, audio and speaking) they find the most helpful.

We were able to find that the most popular learning methods were classes, studying alone with a textbook, and using apps and websites. Classes were preferred because they allowed for conversation practice, however a lot depended on the teacher.

The majority of people said that having context of interest increases their motivation to learn, and that visual data helped them learn and understand. Most also noted that visual & audio materials are the most engaging, and better at helping information retention. For the full set of results, view figure 2 below.

There were a few comments that noted there being a lack of freedom in study material organisation & customisation, which we found interesting

Finally, as shown in figure 3, speaking and practicing was considered most important, closely followed by audio, with videos being the least helpful.

Final comments

We begun the project passionate about focusing on visual context and how this helps learn vocabulary specifically, but this needed to be grounded in a group of people to whom this might benefit. We all acknowledged that this needs to be addressed quickly so we decided to come back to it again with more secondary research.

>>> For my next post on the project, click here.