So Meta: Evaluating KnowledgeNudge
By Masood Khan & Trish Roche
We’ve gone meta — welcome to our blog about evaluating our blog.
More and more research teams are working to incorporate knowledge translation (KT) and patient engagement (PE) in their work. However, evidence shows that resources, tools, and training are needed for teams to be able to practice KT and PE effectively. Online tools, such as blogs, have been found to effective for disseminating evidence-informed information to a broad range of audiences and geographical areas [1].
Our published article provides a great overview of KnowledgeNudge and some of the metrics we’ve collected, but in this post we give you more of the juicy details on how KnowledgeNudge evolved over the years.
This is our story.
The Birth of a Blog
With the unofficial working title “What the Health?”, the idea for a blog about KT was conceived in a conversation over coffee in 2015. Our initial aims were to create an online platform to share innovative and evidence-based information about KT in health research, become a trusted source for KT in Manitoba, and build a community of readers and contributors passionate about KT. We officially launched our blog and Twitter profile as KnowledgeNudge in 2015, but got off to a bit of a rocky start (which we openly admit to in our post about how KnowledgeNudge was our #KTFail).
We bounced back, and managed to post semi-regularly until things changed again in early 2017 — when we picked up and moved our blog from WordPress to Medium. We got into our groove in 2017, when we started consistently posting new content on a weekly basis, becoming more focused in our approach, which evolved to include content around patient engagement (PE) and dissemination of research. We refined our goals, based on performance of different posts and feedback from our audience (collected through annual surveys): to increase awareness, knowledge, and positive attitudes around KT and PE and related best-practice ideas, considerations, and resources.
As our team changed (including our editors and regular contributors) we had to work within our capacity for creating novel content. To emphasize quality over quantity, we moved from weekly to bi-weekly new posts in 2019. We promoted each new blog with at least one Tweet (we give a bit more detail about Tweets per story when we look at our post characteristics).
Where We Are Now
As of July 2020, we sit at over 100 blog posts, 1,000+ Tweets, and 1,500+ combined followers on Medium and Twitter. Topics we’ve covered range from introductions to KT and PE, methods for PE and KT, and KT frameworks, to posts about useful engagement and research approaches, design concepts, interviews, and article summaries. Our contributors (and our team) include a group with a diversity of backgrounds and skill sets, including a KT scientist and trainees, academic researchers, knowledge brokers, a strategic and creative services specialist, and patient engagement professionals.
Measuring Metrics
It’s worth mentioning here that metrics for blogs vary by the platform and host site, but if you have access to the back-end code and some HTML knowledge you can up your analytics game with tools like Google Analytics, which allow you to track additional metrics, links, and even do A/B testing. Having already closed our Wordpress account, we were lucky our editor at the time had the foresight to capture our monthly site visitors and views.
On Medium
Medium, on the other hand, doesn’t provide users with access to HTML code (yet), but they do provide metrics for each day at both the site (publication) and post (story) levels. For a publication (as opposed to individual authors), Medium has a 90-day limit on certain site-level metrics (visitors and minutes read), though you can go back almost indefinitely to collect site views.
If you’re assessing metrics for just one author, Medium has given users even greater access to these and other metrics (including subscribers, fans, and claps — Medium’s version of likes). Authors can go back in 30-day increments to view daily site metrics, but be warned: it takes a lot of clicking.
At the post level, Medium allowed us to go back as far as we wanted to collect individual post views and ‘reads’ by day (where Medium counts a ‘read’ as someone having clicked on a story and kept the page open for the amount of time they estimate it would take to actually read the whole thing).
Since publishing our work, Medium has abandoned the ‘reads’ metric to instead provide views and reading time for each post in graphical form, which you can view for the entire lifetime of the post, or by month.
On Twitter
Twitter is further ahead with their analytics, and much to our delight, saves your metrics beginning with your very first Tweet. Heck, you can even export your data as an Excel file for a specific month (or any period, for up to two months per export). You can also choose to have Twitter sort your data either by Tweet or by day.
We exported data by Tweet for each month of our study period, which provided us with a breakdown of Twitter-specific metrics like impressions (how many times a Tweet is ‘served’ — appearing in someone’s newsfeed) and engagements (how many times someone interacts with a Tweet, including retweets, replies, likes, profile and hashtag clicks, detail expansions, media views, and media engagements).
Selecting Our Metrics
Choosing which metrics to report, and how to describe them, was a challenge and ended up being an iterative and collective process. Using Neiger et al.’s key performance indicators (KPIs) for online activity [2], we landed on a combination of metrics representing exposure, engagement, and reach. The table below compares Neiger’s definitions for each KPI to the metrics we used to address our objectives.
The Study: Retrospective Evaluation of KnowledgeNudge
Alright, back to business. In 2019, we decided to retrospectively analyze how well we were doing in achieving our intended objectives for KnowledgeNudge (both the blog and Twitter profile), over a two-year span from inception (August 2016) to July 2018.
Study Objectives
- Determine changes in exposure and engagement of the KnowledgeNudge blog and Twitter profile; and
- Identify and compare characteristics of blogs posts that had most and least exposure and reach.
Beyond Metrics: Blog Post Characteristics
We also wanted to dig a little deeper to compare how the different characteristics of our posts may have impacted their exposure (views) and reach (reads). The team worked collectively to develop a coding framework to categorize posts published between August 2016 and July 2018, based on the goals and objectives of KnowledgeNudge. Doing so, we came up with two broad categories of post topic and post type, and also included quantitative characteristics in our analysis.
The characteristics we examined for each post included:
- Post topic (KT, PE, both, or other)
- Post type (conceptual, practice, or other)
- Publication date;
- Number of tweets promoting the blog post; and
- Word count.
Objective 1: Site-Level Data
For data at the blog site and Twitter profile levels, we divided our study period into 6-month blocks (four total). In total, between August 2016 and July 2018 we published 99 posts (one per week) by 14 different authors, and 755 Tweets (about 32 a month on average).
If you’re curious, we were pretty darn consistent about posting every week, and only missed four weeks throughout our entire study period — most of them in the first six-month block.
In Figure 1 in the article, we show the exposure and engagement metrics by month. In this post, we break it down graphically by six-month block. For those unfamiliar with the concept, the word ‘significant’ is term used in statistics to describe a difference that is very likely not just due to random chance.
Blog Site Exposure
For our blog site exposure metric, we saw a significant increase in site views per month when we compared the final six-month block to the first three.
Twitter Exposure
Looking at exposure of our Twitter profile, we saw a significant increase in impressions per month when we compared the third and fourth six-month blocks to the first two.
* Significantly greater than block 2 (p = 0.02)
Twitter Engagement
As for engagement with our Twitter profile, we saw a significant increase in engagements per month, but only when we compared the third block to the first.
What Does This Mean?
What we can conclude from these results is that over time, the exposure of our blog site and Twitter profile increased, and this was not due to random chance.
We were surprised to find that our Twitter engagement increased in the third six-month block, but not the fourth — especially since the number of posts and Tweets didn’t change. This indicates there are likely other variables impacting engagement that we didn’t measure. A quick Google search indicates a multitude of possibilities (such as use of hashtags, mentions, the inclusion of images and links, and even the time of day, to mention a few).
Objective 2: Characteristics of Most & Least Popular Blog Posts
We also wanted to know about the performance of individual posts. What makes a KnowledgeNudge post popular (or not)? Using the highest and lowest 10th percentiles for both individual post exposure (average views per week) and reach (average reads per week), we selected seven ‘high profile’ and six ‘low profile’ posts. To see the list of high and low profile posts, check out Table 1 in the full article.
What factors affect whether posts are high or low profile? To try to answer this, we compared characteristics of high and low profile posts. Some characteristics were straightforward (e.g. word count). Other characteristics (post topic and post type) required inductive and deductive coding and categorizing.
Word Count
For KnowledgeNudge, word count was significantly higher for high profile posts than low profile posts.
“How long should a blog post be?” is the first and most common question we get from new authors. If you Google it, you’ll get conflicting answers, though new marketing data indicates posts upwards of 2,000 words are the most popular. In other words, longer is better. Of course, word count alone isn’t the only important factor…
Promotional Tweets
Another factor influencing the performance of a blog is promotion, whether it’s through posting on an organizational website, sharing in newsletters or emails, or social media. Our social media platform of choice is Twitter, known for its popularity among researchers (our key target audience). Interestingly, we didn’t see a significant difference for the number of Tweets per story between high and low profile posts.
Maybe it’s because we’ve been consistent in the number of Tweets we use to promote each post, or maybe because our readers are also referred to stories through shares on other social media platforms (e.g. Facebook), email and instant messaging, engines such as Google and Bing, and to a smaller extent, through the blog site itself.
Time of Publication
Which performs better: older blogs, newer blogs, or maybe something in between? This is an interesting and difficult question to answer, because it involves so many other factors. For example, is an old blog post likely to have more average views and reads because it’s had more time to collect these metrics, or will an older blog be less popular because it’s been buried by newer posts? Perhaps an older blog appears more popular simply because there were fewer blogs to choose from when it was published on KnowledgeNudge. On the other hand, a new post might be more popular due to recency bias and its location on the site (i.e. closer to the top); or because KnowledgeNudge has accumulated more followers over time, providing newer posts with a larger audience base.
All that said, when we compared the time of publishing (using our six-month blocks), we did not actually see any difference between high and low profile posts. Womp womp.
Post Category: Topic and Type
Based on feedback from our audience in our annual reader surveys and previous publication of social media use in disseminating health evidence [3], we decided to ‘code’ each of our 99 blog posts into categories by post type and topic. Through inductive and deductive coding, the team devised a framework for organizing posts by both topic and type.
Post topics include:
- Knowledge translation or KT
- Patient engagement or PE
- Both KT & PE (e.g. event planning, sex- and gender-based analysis)
- Other (e.g. information about the KnowledgeNudge team or the blog itself)
Though most of the high profile posts were about KT (and there were none about both KT and PE, or ‘other’ topics), we didn’t find a significant difference in any of the topics between high and low profile posts. This may mean topic doesn’t matter, but it may also be due to the small number of posts we compared.
Types of posts include:
- Conceptual (providing evidence-based information based on peer-reviewed literature)
- Practice (providing information on tools and resources)
- Opinion (providing expert opinion on KT and PE science and practice issues)
- Other (e.g. providing information about KnowledgeNudge)
Again, though we saw high profile posts having only conceptual and practice posts, and low profile posts including the ‘other’ type, the difference wasn’t statistically significant.
What We’ve Learned
It’s been a long road, and we’ve learned a few lessons along the way. First and foremost, we suggest planning ahead in terms of collecting the metrics you want to capture. It’s going to be a lot easier than going back retrospectively. In our case we were limited by what was available, and missed out on some metrics we would have loved to know about (such as the number of new followers per month, or link directs from Twitter to Medium). In the future, we hope to assess additional factors such as author and guest effects, social media influencer effects, and timing of posts and promotional Tweets. We would also suggest, for those who have the time and initiative, to plan out a social media strategy ahead of time, including creating unique tracking links (through a service like Bitly, for example) for different avenues and social media platforms — or even different Tweet strategies.
Key Blogging Tips from KnowledgeNudge
- Be consistent — post regularly (even if it’s not new content — don’t underestimate the value of recycling or reworking old posts), and try to keep a backlog of posts (or at the very least ideas) you can draw from when other priorities take over;
- Use a scheduling tool like HootSuite or Tweetdeck to plan your social media posts in advance — this way all your sharing can be planned and automated for a week (or more) at once;
- Include aesthetically pleasing visuals wherever possible — nobody likes a wall of text, and you need a catching visual for when you share posts (check out our blogs on visuals for tips and tools);
- Prioritize your current audience — don’t be too focused on gaining new followers; pay attention to what your existing audience is sharing or commenting on, their interests, the type of content they prefer, their needs, and heck — go ahead and ask them what they want;
- Have a call to action — whether it’s “follow us on Twitter”, “read these other blogs”, or “sign up for our email newsletter”, make it easy for your audience to do what you’d like them to do; and
- Don’t be afraid to fail — blogging takes practice. Ask any member of our KnowledgeNudge team of authors, and they’ll cringe when they talk about their first blog. Also, as one of our editors said early on: “your first title will suck”, but that’s what drafts are for.
All in all, KnowledgeNudge has come a long way, and we have many readers, authors, and influencers to thank for their contributions and support (that includes you). It is most definitely a team effort that has helped us grow to where we are today — here’s to increasing awareness of KT and PE in health research!
Want to hear more from KnowledgeNudge? Follow us on Twitter!
For more from CHI, head to our website and sign up for our newsletter, and follow us on Twitter, Facebook, LinkedIn, and YouTube.
