Recognition: Policy and Strategy Impact

If you have visited MindHive this past week you may have noticed our latest feature — MindHive Rankings. The rankings feature uses a tier-based points system to track your growth as a problem solver in the Hive.

Your ranking travels with you around the Hive and is visible when you post comments and join Challenges. It also helps our host organisations when choosing Hivers to invite to their private Challenges.

As it currently stands, you are attributed points for being active in the Hive. The more active you are — the more points you will be accumulating, with certain actions and points required to move up in rank. In the coming weeks we will also be allowing our Challenge and Topic hosts to assign points to those Hivers whose contributions they valued most.

For more information about the Rankings, and to see who our leading problem solvers are, click the purple button below. Visit your profile page to see which level you are currently aligned with. To help those who want to move up from Novice to Hiver:

  1. Make sure you have logged into the MindHive website at least four times.
  2. Ensure that your MindHive profile is complete. Yes that means a profile picture too!
  3. Join at least one MindHive Challenge. If you haven’t yet been invited to participate in a Challenge you can find the publicly listed ones here. Open the Solution tab on one that meets your fancy and accept the invitation to join.
  4. You need to post at least two comments to make your way to Hiver status. These comments can be made against a Challenge or a Topic. You can find a list of Topics here.

Increasing the engagement of MindHive contributors

In principle, MindHive helps government via contributions to important policy issues from a broad base of experts drawn from a variety of fields., a innovative and practical method for policy and strategy development that cuts through the complexity of broad consultation by the use of MindHive’s technology. MindHive also offers universities opportunities to build relationships with other organisations outside of academia and demonstrate their research impact and enables Not-for-profit organisations (NFPs) to get ahead of the pack and profile policy issues directly in front of those who need to see it.

More generally, MindHive aims to facilitate partnership, research and collaboration between industry, government and universities and to help partners share, promote and profile their organisational research as well as individual researchers via policy contribution and exchange opportunities. Given the above, it is important to able to measure MindHive’s impact by using metrics that focus on organisational research and individual researchers.

At present, and in contrast with the Facebook (https://www.facebook.com) and Kaggle (https://www.kaggle.com/) websites, there is potential to grow the potential pool of MindHive contributors engages in challenges. Given that the aim of the MindHive site is to bring a range of informed views to bear on challenges, we’re implementing our new ranking feature.

Facebook users are numerous and sufficiently engaged to refresh the viewing pages many times daily. These users are in the main reinforced in their engagement with Facebook by liking and sharing images, text and URLs. Kaggle is also a site with numerous users engaged in competitions involving data prediction via coding (e.g., writing elegant algorithms) or decoding (e.g., interpreting data flows via figures, etc.). Kaggle users are reinforced in their engagement via prizes for problem solving, by user ratings, and also by being rated on a three-step Mastery scale (Novice, Kaggler, Master), as follows:

  • Novice: A new member on the site
  • Kaggler: This is the widest tier, representing the main body of the Kaggle community
  • Master: To achieve this tier, one must demonstrate both consistency and excellence.

At the moment, MindHive website has the capacity to increase client engagement by providing analytic counts of impact based on interactions such as the number of users confirming they want to contribute, users voting, users viewing issues pages, users involved in panels, users viewing policy pages and responding to policies, users viewing events pages and viewing and responding to policy outcome pages. Beyond this, MindHive now measures the impact of user engagement by rating users on a four-step scale equivalent to the Kaggle Master scale: Novice, Hiver, Master, All Star.

An issue the Development Team have worked on (aside from how a contributor could be said to have ‘solved’ a challenge) is how to encourage higher levels of participation from potential contributors and how to reward contributors for their participation.

A first step to greater engagement was to increase participation levels. Options to increase participation included:

  • All potential contributors notified when new challenges become available
  • Confirmed contributors emailed regarding additional comments
  • The email link takes them directly to comments (instead of a series of login pages)
  • The comments are embedded in text itself (rather than in sidebars as at present).

The next step involved options to reward participation using some type of user evaluation of comments. With that in mind, Idler & Kasl (1991) have reported self-evaluations of health to be a significant predictor of mortality in follow-up studies. That is, above and beyond several other different indicators of health, such subjective perceptions carry an independent ability to predict the individual’s survival over a period of as many as six to nine years. Based on Idler & Kasl’s conclusions, user ratings of MindHive comments based on specific Likert scale items can be very predictive of future performance. Users rate one another’s comments on one or more five-point scales, such as:

· Relevance: Not necessarily useful, slightly useful, moderately useful, very useful, highly useful

· Expertise: Intermediate expertise, advanced expertise, highly accomplished, master, grand-master

· Likeability: Neither like nor dislike, like slightly, like moderately, like very much, like extremely.

A rationale for not including explicitly negative ratings is that negative ratings are an invitation to trolling that discourages brainstorming, where brainstorming is an essential intermediate step if quality challenge outcomes are to be based on collated comments.

In time, MindHive would reward its users for participation by building reputations based on a combination of frequency of comments and the average level of likability, expertise or relevance of comments, as rated by other users. Since reputation would be based on frequency as well as the likability, expertise or relevance of comments, a single highly rated comment would not suffice. The MindHive website will provide a reputation page that lists contributors by the average frequency of their comments across challenges as per Kaggle’s user rankings (https://www.kaggle.com/users). The MindHive website will also generate a Mastery scale based on some composite of frequency and consistency of comments, as per Kaggle’s Mastery scale (https://www.kaggle.com/wiki/UserRankingAndTierSystem).

The engagement of our Ranking measure encourages more frequent high quality comments that enhance challenge outcomes but also a sense of community amongst users.[1] More generally, these metrics allow MindHive to measure the impact of their methodology both in terms of organisational research (via analytic counts) and individual researchers (via earned reputation).

References

Ellen L. Idler1 and Stanislav Kasl (1991). Health Perceptions and Survival: Do Global Evaluations of Health Status Really Predict Mortality? Journal of Gerontology: SOCIAL SCIENCES, Vol. 46, №2, S55–65

[1] Both Facebook & Kaggle include messaging options that could be relevant at some point for MindHive contributors as well.


Originally published at www.mindhive.org on 12 October, 2016. With thanks to Peter Grimbeek, Eidos Associate (2006-) for his significant contribution to the methodology design implemented.

Join this discussion in the Hive


Bruce Muirhead is the CEO of Eidos Institute and the Founder of MindHive.org — More Blogs here: https://mindhive.org/slp/blog.html