Chatterbug’s beta as a technical power user

The shortest path to real-life conversations

It started with “GitHub’s scandalized ex-CEO returns with Chatterbug” from Techcrunch, while I was surfing Google Chrome’s news showcased on the “New Tab” screen on my mobile. I read the story, and am a fan of GitHub for it’s simplicity, design, and functionality. My association with it was with quality, so I decided to check the site out. In a time filled with Trump, Harvey Weinstein, George H.W. Bush; riding the ripples of allegations and assaults with a powerful word like “scandalized” was one way to get publicity.

Chatterbug aims to be the complete solution to language learning by employing 45 minute live lessons for listening and speaking, and use of machine learning for “spaced repetition” timing, and much more. I decided to entrust myself and time to this AI with the hopes of some language proficiency regardless of language, within their free 30 day tier before terms changed(I was an active Cerego user, up until they stopped publishing public content and started focusing on businesses) Chatterbug also has business pricings, which could lead to similar business decision as Cerego. Cofounder/CEO Scott Chacon’s blog post includes a video titled “Learn German in 150 Hours” which is enticing to those who don’t read article which also states that language learning is a marathon. The 150 hours is of live lessons, which also includes 150 hours of study for a total of 300 hours within the time frame of 8 months. I figured if spaced repetition worked on a micro scale, then it should work on a macro scale allowing me to spend most of the day studying and improving at a rate much faster.

I signed up and started the lessons, while enjoying the simple user interface. The only language is German(with more to come), which is somewhat relatable to English and somewhat straightforward to learn. Pronunciation is tough, and without someone to correct you all the time, listening and repeating the sounds is a must so you don’t perpetuate/practice erroneous sounds. Memorizing German words with visual associations is quite easy and fun, with only a few hard words so far like verstehen, which I kept on getting wrong until I had more words memorized to notice the spelling patterns. Onward I went for 5 days accumulating ~8 hours of tracked study, and 190 new words. Having messaged Scott, the site was not yet read for tablet and mobile, so I bought a best value Logitech C270 HD webcam. I joined a few random German video chat rooms and found a willing Lebanese partner (who spoke a bit of English and German, French and Arabic) living in Germany as refugee. We chatted in German and English for 20 minutes, sharing tips/resources and experiences. He said English could be used in the streets among other refugees, but Germans would only reply in German. This was bad for him and his situation where communication is important, but could be good for those who wish to be immersed in the language. At the end of this chat I was really excited to try my first live lesson on Chatterbug, which was unlocked after reaching a certain proficiency(2 days for me).

As a technical user, I enjoy being curious and providing actionable feedback to developers with regards to the user interface, user interaction, and functionality. Chatterbug has a feedback chat powered by Intercom and I happily used it. My first message was about image consistency and first impressions, and engagement opportunities with regards to content personalization based on user location/country. I received a response thanking me for the suggestion, and following up on one of the issues. My second message was about a user experience optimization, where after completing a session, I would have to go through 3 screens to repeat a lesson, or explicitly choose another(they have an implicit option). Intercom allows for the team to answer questions, but this question has yet to be addressed which leads to a bit an analysis.

Answering questions, and replying requires a classification of the question. How valid is the question? How important is the question? How important is acknowledging the user? To what depth should the acknowledgment be? How timely is a timely reply? Who is best able to answer the question? What do we do if we don’t have someone in the position/expertise to answer the question? How often do we want our experts answering the questions? How do we answer a question that we have no technical power over? What is the risk of answering the question inappropriately? What is the business/technical priority for acting on the feedback? How do we parse and document a worthy question that is multi part? How do we respond to a multi part question without making the user feel like some of the concerns were ignored? How do we recognize the cause, when given the symptoms? How do we prevent misclassification bias(leading to ignored problems that are relevant to other parts of the organization)? What kind of organizational crosstalk is need to effectively address a question? What is the process of addressing a complex question without causing organizational slowdown? What causes others to ignore questions(always having a goto person, not being having relevant skill to the user’s query)?

My third question was about previewing the lessons(minor feature) for the live lesson(key functionality). The response was timely, occurring before the lesson, but it would be important to state that if that feature had not worked correctly, a fix may or may not be available before the appointment, thus improper functionality should be expected. My fourth question was about reaching a milestone but still lacking an understanding of why certain words were capitalized within the sentence. This is rather ambiguous, it could be due to my lack of effort, or due to the systems poor implementation. I was happy to receive a personalized message clarifying the issue. However, that only solves my issue. As an aside: at scale, there could be several people with the same issue, thus requiring a feature analysis. How a domain expert communicates this issue forward will affect a large segment of the userbase. This is why it is important to make each employee feel empowered and capable to work across organizational boundaries and concerns, else you have to silo-ed viewpoints. My fifth concern was about a calendar sync problem, to which the reply contained technical terminology providing a very clear response and a showcase of competence. Know your user and respond appropriately, too much or too little can lead to a disconnect with the brand. My sixth message was about an unintended usage case, where I incremented the lesson numbers within the URL, causing my curriculum/system to ask questions well beyond my abilities(this problem snowballed into a larger one during my live lesson). I thought of this bug partially thanks to a security meetup which described the basics sql injection/data mining, and due to language/progression curiosity. This would be a great time to investigate the need for a feature that showcases more advanced users live lessons, or something that gives new users some kind of excitement for the future. It also shows how a system can be corrupted by unexpected engagement thus requiring manual/automated efforts to rectify the issue. Having safeguards against this kind of manipulation/gamification could be critical when catering to businesses that require certifications awarded by your company, and may put the validity of your organization’s authority at risk. How do we reply to such an offender, thank them for their efforts? ignore them and the account’s state? offer to fix the problem? how much money will fixing one person's problem cost? how do we respond if we can’t/don’t want to fix it? more questions to consider.

My latest interaction and message was with regards to the first live lesson I attended as a student. I was using a real microphone, to allow my partner to really hear my pronunciation instead of relying on webcam audio. I ran into UI bugs, critical buttons where hidden behind the video feed. Audio had loopback problems due to two tabs being open on the partners browser. I could not talk properly due to the feedback, nor was I confident with the full material being displayed during the lesson(was I missing an important piece of the lesson? I am using a 4k monitor with more than enough screen resolution). The ghosts from past tinkering showed up as part of the lesson plan causing it to be too difficult, but I had managed to skip them and I accept responsibility for that problem. Thankfully, my partner was very helpful, danke Antje, and switched to a more ad-hoc approach to the lesson. I also had difficulty listening due to my partners microphone audio level auto adjust, which I guess is a software thing(it looked like she was using Apple headphones) some of the words would be cut off, which is not ok when you are learning a language.

Review past live lessons

That said, coordinating live video, video statistics, recording mouse movements, organizing two different clients(teacher/student), scheduling, is no easy task. I understand that it will take time to get right, and nor do I expect to get a free lesson/compensation due to the troubles unless I was paying(Chatterbug is about the centralization of language learning, pay for as many lessons as you need, or based on your subscription model) I would have liked some acknowledgment of the many issues stated in the feedback, especially when they were technical problems, but I can only imagine that the team is at work trying to fix these problems. Perhaps a response is kommen soon.

Update: Chatterbug has fixes for both the URL issue(changed lesson numbers URL to encoded alphanumeric string) and also the multiple video issue(via a warning), and I get to try the live lesson again 👍