Building Twatter: Gathering requirements

Previously on ‘Building Twatter’…

Building Twatter: Purpose

This post will be lengthier than normal as there’s a lot to go through. It’s always tempting to just jump into coding and figure stuff out as you go along, but defining actual requirements will really help speed up development in the long term.

There are a set of fundamental features that I know we’ll definitely need for Twatter. I’ll split them up into functional and nonfunctional requirements.

Functional requirements

• User authentication
• Create a profile
• Compose a twot
• Like a twot
• Share a twot
• Reply to a twot
• Follow a twat

Nonfunctional requirements

• The dashboard
• Notification centre
• Profile page
• Data visualisation of trending subjects
• Natural language processor to determine subject

I’m going to study existing web apps that shares Twatter’s goal of instantly sharing content with the public. I’ll also look at web apps that visualise data from microblogging services and compile a list of requirements for Twatter. I’ll then split this list into further lists using the MSCW (Must-have, Should-have, Could-have and Would-have) method.

I will be defining an MVP (minimum viable product) based on the must-haves and layout a plan for when the should-, could- and would-haves can be implemented.

An open source microblogging platform with data visualisation on trends

The web apps I’ll be studying and comparing are:

• Twitter
• Mastodon
• Medium

User authentication and creating a profile

I don’t think this needs much research as it’s a well thought-out mandatory feature for all web apps. However, to simplify things I’ll use the Twitter OAuth API to authenticate a user and create their profile.

Resources

• Sign in with Twitter

Compose a twot

This is probably the most important feature so I’ll go into more detail on my analysis of the existing platforms.

Tweet - Twitter

As you can see, Twitter has a lot of features that go with composing a tweet and the famous limitation of 140 characters. You can enter text, hashtags, links, user links, images, polls and location data.

I think we can all agree that the text area and a submit button is definitely needed.

Toot - Mastodon

Mastodon has a very similar flow to Twitter with different limitations. You can’t attach polls or a location but you have a larger character limit of 500. Hashtags, links and user links work in the same way but don’t have the aesthetic of Twitter.

Story - Medium

Medium isn’t really a microblogging platform but I like the UX and design, so i’ll be using it as a source of inspiration for Twatter. You can add images, links and tags to stories.

Summary

The common features allow users to embed links, images, tags and links to other users. Twitter and Mastodon enforce a character limit, which distinguishes it as a microblogging platform. Twatter will also need a character limit to be considered a microblogging platform.

Twitter uses a URL shortener for links to give users back some characters for their tweet. This feature is very useful and one I plan to incorporate in Twatter. They also use the target URL’s metadata to attach a snippet to the tweet. A nice looking feature, probably not a must-have, but definitely worth considering to implement.

In comparison, Mastodon takes a leaner approach and implements neither a URL shortener or snippet.

One would assume that hashtags are a standard feature for a microblogging platform, but they can be misused and are prone to human error. We’ll be replacing this feature with automatic tagging of twots using a natural language processor.

Therefore, the ‘Compose a twot’ requirements for Twatter are as follows:

• Allow users to enter text with a 210 character limit
• Display live character count
• Allow users to post a twot by clicking on a button
• Allow users to embed URLs and automatically shorten using Bit.ly’s API
• Allow users to tag twats by using ‘@<username>’
• Automatically highlight and link URLs (http://) and twats (@).
• Allow users to attach an image
• Validate twots when the text area loses focus and before posting a twot

Like a twot

This one’s easy. They all have a very similar approach to liking posts. We’ll continue to use the standard.

• Users can like a twot by clicking on it’s like icon button
• There will be a live count of likes displayed next to the like icon button

Share a twot

Again, this is another obvious feature that can only really be done in one way. The idea here is to share a comment from another user that you think your followers may find useful.

Twitter allows you to add a comment to a Retweet, which turns the shared tweet into a snippet that is attached to your comment.

Mastodon simply posts the Toot to your feed and Medium only prompts you to share a story on Twitter or Facebook.

We’ll refrain from the Twitter route and do a simple re-post of the twot.

• Users can share a twot to their followers by clicking on the share icon button
• There will be a live count of shares next to the share icon button

Reply to a twot

All three platforms allow users to reply to a post. Twitter and Mastodon reference the post in the reply. Medium is more akin to a traditional blogging system where users can comment on a post.

It’s a nice feature that builds threads of conversations around a twot. In terms of development, it would require the addition of a reference to the twot, the ability to fetch replies and view it as a list.

• Allow users to reply to a twot by clicking on the reply button
• Allow users to compose a reply using the same functionality as composing a twot (minus the mandatory user link)

Follow a twat

Quite self-explanatory. Each platform has a follow/unfollow button on a users profile. Following a user means you receive all their posts in your feed.

• Allow twats to follow other twats
• Allow twats to unfollow a twat
• When you follow a twat you will receive their twots in your feed
• Allow users to view a list of the twats they follow
• Allow users to view a list of the twats that follow them

The dashboard

The feed is another important feature that demands more attention in its analysis. Again, I’ll look at how each platform tackles this feature.

Twitter

There’s a lot happening on Twitter’s dashboard. They’ve split up the page into 3 columns. The left having a mini profile view of the logged in user and trending hashtags, the centre containing the personalised feed and finally the right offering suggestions for people to follow and company info.

The feed has a form above it to compose a tweet, which is good UX, and all the tweets are chronologically ordered showing the latest first.

Mastodon

Mastodon has a (surprise, surprise) similar approach to Twitter, although it looks a lot more like TweetDeck.

There’s a quick way to compose a toot and 3 different feeds: a personalised, notification and public feed.

The personalised feed will contain toots from people you follow and your own. The notification feed displays new followers and any interactions made by other users to your toots i.e. replies, retoots and likes.

The public feed displays everyone’s toots in the network. This can be change to geographically local and global users/toots.

Medium

Medium displays a feed of stories of users and tags you follow, and lists the tags you follow beside it. Quite straightforward.

Summary

The most common feature on a dashboard is the personalised feed displayed in chronological order (latest first). Each post displayed in the feed allows you to interact with it (reply, share and like).

When you click on a post you get a more detailed view. Twitter opens it in a modal, Mastodon opens it in the far right column and Medium goes to a new page. In detailed views you get to see a list of all the replies to the post too. Mastodon also displays snippets to links and tagged users like Twitter does.

The core requirements for the dashboard are as follows:

• Display a personalised feed of twots from twats the user follows
• Allow the user to interact with twots in the feed (reply, share and like)
• Allow users to view a twot in more detail including and replies
• Allow users to easily compose a twot
• Display subject trends
• Display a compact profile card of the logged in user

Notification Centre

The notification centre is effectively a feed of interactions from other users that concern you. If anyone likes, shares or replies to one of your posts or someone follows you, you will receive a notification.

Twitter have a page for their notification feed. They categorise and group similar notifications.

Sadly, I’ve not received any notifications on Mastodon but having searched on Google Images, it seems they do not group notifications and display each interaction separately.

Medium’s is simple and drops down from the nav bar. It displays each interaction separately.

• Allow users to view a feed of notifications
• Indicate which notifications are new
• Alert the user to a new notification
• Allow users to go to the source of the interaction from the notification

Profile page

All three platforms have a similar approach to the profile page. This feature is easy to define:

• Display user profile image
• Display user name
• Display user bio
• Display number of followers
• Display number of twats being followed
• Display follow/unfollow button
• Display all twots of user in a feed

Data visualisation of trending subject

Medium

The Stats section on Medium is a useful tool for bloggers, giving them insight to the interactions with their stories. The numbers are visualised in a minimalist bar chart. Hovering over a bar displays the exact number and date. You can filter the chart by views, reads and recommendations.

TweetBinder

I couldn’t find any visualisation of data or statistics on Twitter and Mastodon, so looked to a 3rd party web app that does. TweetBinder takes a hashtag and provides a generous collection of insight.

There are a few good ideas here that we could borrow, but we would base them around trending subjects derived from the NLP (natural language processing) module we end up using.

So, for data visualisation the requirements would be:

• Display a list of trending subjects
• Allow users to to view detailed statistics of a trending subject
• Display total number of twots made about subject
• Display total number of shared twots about subject
• Display total number of replies made about subject
• Display total number of twats that participated in subject

Natural language processor to determine subject

I looked at 3 different Node implementations of natural language processors, each one having pros and cons, some more advanced than others. I won’t go into the details of NLP, but you can find and read many publications on the subject, online.

What we want to achieve with NLP is to be able to determine the subject matter of a post and whether the comment is positive or negative about the subject. In the NLP world these functions are called classification and sentiment.

Natural

NaturalNode/natural

The first library by NaturlaNode is quite advanced and has a lot more functions than required.

Tokenizing, stemming, classification, phonetics, tf-idf, WordNet, string similarity, and some inflections are currently supported.

Reading about the classification function in NaturalNode, it seems you have to train the processor for it to be able to determine what the action is of a string.

In all honesty this went over my head. I haven’t studied natural language processing nor do I intend to. I simply require to use it as an abstract function. NaturalNode is very advanced and requires a developer to know and understand the theories of NLP.

So, I decided to not go with NaturalNode simply for the fact that it requires further learning and investigation.

Sentiment

thisandagain/sentiment

In my search for a sentiment based processor, I came across Sentiment by thisandagain. It’s a simple library that “does what it says on the tin” by providing a score on whether the string has a positive or negative sentiment.

This is great because it doesn’t require me to understand how the sentiment score is calculated, just that it uses an official industry standard to determine the result.

Here’s an example of its usage:

var sentiment = require('sentiment');

var r1 = sentiment('Cats are stupid.');
console.dir(r1);        // Score: -2, Comparative: -0.666

var r2 = sentiment('Cats are totally amazing!');
console.dir(r2);        // Score: 4, Comparative: 1

This Node module helps identify sentiment but we still need to determine the subject matter of a string. This led me to discover SpeakEasy.

SpeakEasy

nhunzaker/speakeasy

I was pretty happy to find SpeakEasy as it had the features I needed, classification and sentiment.

I had to test out Sentiment and SpeakEasy to see what sort of results they would return, and if they fulfilled my objective of determining subject matter and sentiment.

I grabbed various tweets from Twitter from a trending subject that day, put them in an array and ran it through the two nlp libraries.

const twots = [
  'Sadiq Khan has withdrawn his support for the Garden Bridge project.',
  'BREAKING: Huge blow for Garden Bridge as Mayor refuses  guarantees. Says London tax payer would be left with "too much financial risk"',
  'Garden Bridge gone! Now we should investigate the flawed procurement process that allowed £60m to be wasted.',
  'Garden Bridge scuppered by a muslim mayor must be a bitch for the daily mail they cant know who to blame',
  'Ding dong, the Garden Bridge is dead.',
  'Hooray that the  poncy, ridiculous, waste-of-money garden bridge is not going ahead. About time this decision was made. Thank you Sadiq Khan.',
  'Local squirrel disappointed that garden bridge is to be scrapped because he heard everyone saying the bridge was nuts.',
  'Good job the garden bridge has been cancelled. Poor dears would have exterminated the trees...'
];
// Sentiment
let sentiment = require('sentiment');
// SpeakEasy
let speak = require('speakeasy-nlp');
let subjects = [];
for (var i = 0; i < twots.length; i++) {
  console.log('--------------------------------');
  console.log(twots[i]);
  console.log('---------------');
  console.log('Sentiment');
  let senti = sentiment(twots[i]);
  console.log('score: ', senti.score);
  console.log('comp: ', senti.comparative);
  console.log('positive: ', senti.positive);
  console.log('negative: ', senti.negative);
  console.log('---------------');
  console.log('SpeakEasy');
  let classy = speak.classify(twots[i].toLowerCase());
  let speaky = speak.sentiment.analyze(twots[i]);
  subjects.push(classy.subject);
  console.log('action: ', classy.action);
  console.log('owner: ', classy.owner);
  console.log('subject: ', classy.subject);
  console.log('score: ', speaky.score);
  console.log('comp: ', speaky.comparative);
  console.log('positive: ', speaky.positive);
  console.log('negative: ', speaky.negative);
}
console.log(subjects);
let subjects = [];
for (var i = 0; i < twots.length; i++) {
  console.log('--------------------------------');
  console.log(twots[i]);
  console.log('---------------');
  console.log('Sentiment');
  let senti = sentiment(twots[i]);
  console.log('score: ', senti.score);
  console.log('comp: ', senti.comparative);
  console.log('positive: ', senti.positive);
  console.log('negative: ', senti.negative);
  console.log('---------------');
  console.log('SpeakEasy');
  let classy = speak.classify(twots[i].toLowerCase());
  let speaky = speak.sentiment.analyze(twots[i]);
  subjects.push(classy.subject);
  console.log('action: ', classy.action);
  console.log('owner: ', classy.owner);
  console.log('subject: ', classy.subject);
  console.log('score: ', speaky.score);
  console.log('comp: ', speaky.comparative);
  console.log('positive: ', speaky.positive);
  console.log('negative: ', speaky.negative);
}
console.log(subjects);

Here were the results:

--------------------------------
Sadiq Khan has withdrawn his support for the Garden Bridge project.
---------------
Sentiment
score:  2
comp:  0.18181818181818182
positive:  [ 'support' ]
negative:  []
---------------
SpeakEasy
action:  sadiq
owner:  his
subject:  support
score:  1
comp:  0.09090909090909091
positive:  { score: 1,
comparative: 0.09090909090909091,
words: [ 'support' ] }
negative:  { score: 0, comparative: 0, words: [] }
--------------------------------
BREAKING: Huge blow for Garden Bridge as Mayor refuses  guarantees. Says London tax payer would be left with "too much financial risk"
---------------
Sentiment
score:  -3
comp:  -0.13043478260869565
positive:  [ 'huge' ]
negative:  [ 'risk', 'refuses' ]
---------------
SpeakEasy
action:  be
owner:  guarantees
subject:  breaking: huge blow garden bridge mayor refuses guarantees says london tax payer would be left
score:  -2
comp:  -0.08695652173913043
positive:  { score: 0, comparative: 0, words: [] }
negative:  { score: 2,
comparative: 0.08695652173913043,
words: [ 'blow', 'refuses' ] }
--------------------------------
Garden Bridge gone! Now we should investigate the flawed procurement process that allowed £60m to be wasted.
---------------
Sentiment
score:  -5
comp:  -0.29411764705882354
positive:  []
negative:  [ 'wasted', 'flawed' ]
---------------
SpeakEasy
action:  investigate
owner:  flawed procurement process that allowed £ 60 m to wasted
subject:  flawed procurement process
score:  -1
comp:  -0.058823529411764705
positive:  { score: 0, comparative: 0, words: [] }
negative:  { score: 1,
comparative: 0.058823529411764705,
words: [ 'flawed' ] }
--------------------------------
Garden Bridge scuppered by a muslim mayor must be a bitch for the daily mail they cant know who to blame
---------------
Sentiment
score:  -7
comp:  -0.3333333333333333
positive:  []
negative:  [ 'blame', 'bitch' ]
---------------
SpeakEasy
action:  who
owner:  undefined
subject:  muslim mayor must be a bitch
score:  -2
comp:  -0.09523809523809523
positive:  { score: 0, comparative: 0, words: [] }
negative:  { score: 2,
comparative: 0.09523809523809523,
words: [ 'bitch', 'blame' ] }
--------------------------------
Ding dong, the Garden Bridge is dead.
---------------
Sentiment
score:  -3
comp:  -0.42857142857142855
positive:  []
negative:  [ 'dead' ]
---------------
SpeakEasy
action:  dead
owner:  dead
subject:  garden bridge
score:  -1
comp:  -0.14285714285714285
positive:  { score: 0, comparative: 0, words: [] }
negative:  { score: 1, comparative: 0.14285714285714285, words: [ 'ding' ] }
--------------------------------
Hooray that the  poncy, ridiculous, waste-of-money garden bridge is not going ahead. About time this decision was made. Thank you Sadiq Khan.
---------------
Sentiment
score:  -1
comp:  -0.043478260869565216
positive:  [ 'thank' ]
negative:  [ 'ridiculous' ]
---------------
SpeakEasy
action:  thank
owner:  listener
subject:  poncy , ridiculous , waste-of-money garden bridge is not going ahead
score:  2
comp:  0.08695652173913043
positive:  { score: 2,
comparative: 0.08695652173913043,
words: [ 'hooray', 'thank' ] }
negative:  { score: 0, comparative: 0, words: [] }
--------------------------------
Local squirrel disappointed that garden bridge is to be scrapped because he heard everyone saying the bridge was nuts.
---------------
Sentiment
score:  -5
comp:  -0.2631578947368421
positive:  []
negative:  [ 'nuts', 'disappointed' ]
---------------
SpeakEasy
action:  be
owner:  undefined
subject:  he
score:  -1
comp:  -0.05263157894736842
positive:  { score: 0, comparative: 0, words: [] }
negative:  { score: 1,
comparative: 0.05263157894736842,
words: [ 'disappointed' ] }
--------------------------------
Good job the garden bridge has been cancelled. Poor dears would have exterminated the trees...
---------------
Sentiment
score:  0
comp:  0
positive:  [ 'good' ]
negative:  [ 'poor', 'cancelled' ]
---------------
SpeakEasy
action:  good
owner:  been cancelled
subject:  garden bridge
score:  0
comp:  0
positive:  { score: 1, comparative: 0.06666666666666667, words: [ 'good' ] }
negative:  { score: 1, comparative: 0.06666666666666667, words: [ 'poor' ] }
[ 'support',
'breaking: huge blow garden bridge mayor refuses guarantees says london tax payer would be left',
'flawed procurement process',
'muslim mayor must be a bitch',
'garden bridge',
'poncy , ridiculous , waste-of-money garden bridge is not going ahead',
'he',
'garden bridge' ]

The results were a bit weird. Obviously, these modules are not state-of-the-art NLPs. They are basic in design.

Overall, Sentiment did return better sentiments, which shouldn’t be a surprise, but SpeakEasy was hit and miss on determining subject. This could be down to my understanding of subject in NLP, but the results improved when I converted all the strings to lower case. It seemed to ignore capitalised words, deliberately, as subjects. I think because it assumes it’s a name or place, which is correct, but I do not know the reason why a name/place can’t be a subject.

I think using a mixture of both NLP modules will give me the desired results or at least something close enough.

Fuck me, that’s a long post

If you’ve made it this far, I can tell you that I feel your pain. If you think it was bad enough reading through this long post, imagine how it was writing it!

Despite that, I have learnt a thing or two about microblogging services and I feel I know exactly what features Twatter will have.

Now here’s a big fuck off list of all the requirements I’ve gathered of the above with either M, S, C or W next to it.

Reminder: MSCW = Must-have, Should-have, Could-have and Would-have

User authentication

• Implement Twitter OAuth API — M

Create user profile

• Use Twitter profile data to populate profile — M

Compose a twot

• Allow users to enter text with a 210 character limit — M
• Display live character count — M
• Allow users to post a twot by clicking on a button — M
• Allow users to embed URLs and automatically shorten using Bit.ly’s API — S
• Allow users to tag twats by using ‘@<username>’ — M
• Automatically highlight and link URLs (http://) and twats (@). — M
• Allow users to attach an image — S
• Validate twots when the text area loses focus and before posting a twot — M

Like a twot

• Users can like a twot by clicking on it’s like icon button — M
• There will be a live count of likes displayed next to the like icon button — M

Share of twot

• Users can share a twot to their followers by clicking on the share icon button — M
• There will be a live count of shares next to the share icon button — M

Reply to a twot

• Allow users to reply to a twot by clicking on the reply button — M
• Allow users to compose a reply using the same functionality as composing a twot (minus the mandatory user link) — M

Follow a twat

• Allow twats to follow other twats — M
• Allow twats to unfollow a twat — M
• When you follow a twat you will receive their twots in your feed — M
• Allow users to view a list of the twats they follow — M
• Allow users to view a list of the twats that follow them — M

The dashboard

• Display a personalised feed of twots from twats the user follows — M
• Allow the user to interact with twots in the feed (reply, share and like) — M
• Allow users to view a twot in more detail including and replies — S
• Allow users to easily compose a twot — M
• Display subject trends — M
• Display a compact profile card of the logged in user — M

Notification centre

• Allow users to view a feed of notifications — M
• Indicate which notifications are new — M
• Alert the user to a new notification — M
• Allow users to go to the source of the interaction from the notification — M

Profile page

• Display user profile image — M
• Display user name — M
• Display user bio — M
• Display number of followers — M
• Display number of twats being followed — M
• Display follow/unfollow button — M
• Display all twots of user in a feed — M

Data visualisation of trending subject

• Display a list of trending subjects — M
• Allow users to to view detailed statistics of a trending subject — S
• Display total number of twots made about subject — M
• Display total number of shared twots about subject — M
• Display total number of replies made about subject — M
• Display total number of twats that participated in subject — M

Natural language processor to determine subject

• Determine the dominant subject of a twot — M
• Determine the sentiment of a twot (is it positive or negative?) — M

Most of the requirements are must-haves, which is always the case on the first round. This will change as we start fleshing out the requirements. We’ll realise that some features are really not necessary for the app to function well and still achieves it’s goal. To be fair, we haven’t really defined in detail how these features will be achieved, but when we do we’ll end up with the big bells and whistles version, and the lean version. Our MVP will always pick the lean version of each feature and build that.

The next post should be a bit more fun as I’ll be drawing up wireframes and shaping up the user flow of Twatter. The UX will give us a clear idea of what we can cut back on in development.