How video affects page performance: Results from our pilot project
Key statistics and takeaways from our work to support and improve Mass.gov content with video
In the fall, we wrote about a new pilot project for Mass.gov that focused on how video affects page performance. We know that any comprehensive modern digital strategy has to include video. Customers expect it. The question was, how can we optimize video within the Mass.gov environment?
For this pilot, we produced nine videos to be posted on Mass.gov for nine different Commonwealth agencies, with topics of focus ranging from where to find a park for the Department of Conservation and Recreation, to how to report child abuse for the Department of Children and Families. The goal was to explain an important or complicated process in order to address certain frustrations that customers experienced, translate a text-heavy page for users who may not have time to read the whole thing, or to encourage a specific action.
We published each video on a service page, the backbone content type of our service-focused strategy for Mass.gov, and included a clickable call-to-action link at the end of each video. We chose to host the videos in Vimeo because they offered this clickable link function. The video module was placed near the top of the page, below the large headline and right beside the overview text.
What We Found
In the first two months after publication, our videos received 3,309 views and 1,017 completions (meaning the user kept the video playing until the end) — a 30.73 percent completion rate. After hitting play, users watched an average of 68 percent of the video. When compared against the total impressions for the pages where videos were hosted, our videos had an average watch rate of 3.32 percent.
This data was good to have, but we needed a baseline set of metrics to compare them to in order to effectively analyze each video’s performance, and the performance of the pilot as a whole. Because we’re usually not working towards a specific sales goal, private sector standards and best practices don’t completely align with what we were trying to achieve, and we didn’t have a good way to compare our numbers with other public sector organizations.
So we decided to compare our videos to data we could pull from other videos that were already posted on Mass.gov. Because the majority of these videos were not created by Digital Services, we had no record of their individual topics, lengths, or quality, but the data provided gave us a good baseline to determine whether our pilot strategy could be successful in the Mass.gov environment.
The results were encouraging. The average video on Mass.gov has a completion rate of 9.54 percent — meaning that the completion rate for the nine videos we produced was three times the average for the site. We see this as a good indicator that the target length of our videos (less than two minutes) is effective, and that our audiences are responding to each video’s pace, branding elements, and information.
We also asked each customer if they had any metrics they were measuring that our group might not necessarily be aware of or have access to that might help us evaluate their video’s performance. The most interesting metric we received was from the Massachusetts Division of Fisheries and Wildlife (MassWildlife). With MassWildlife we produced a video focused on their Hunter Education Program, which hunters may need to complete in order to obtain their first hunting license. We ended the video with a clickable link to sign up to get notifications of upcoming courses.
We front-loaded the video with the most pertinent information to the audience: who needs to take this course in order to get a hunting license, and who already qualifies for a license without the course. Following up with MassWildlife after the pilot completed, we learned that sign ups for course notifications notably dropped since the video was published, both compared to the previous quarter and the previous year. This initially concerned our team, and we started to brainstorm why that might be.
MassWildlife had a different take. Based on their experience and what they heard from constituents, people watched the video and learned that they already qualified to get a hunting license without having to take the hunter’s education course. The drop cut down on people signing up for a course they didn’t need. It was a good thing. While we’re not certain that the video is responsible for the decrease in sign-ups, the data suggests that this is happening. MassWildlife and Mass.gov will continue to monitor user behavior over the entire year to get a clearer picture of the effect the video is having. The experience reiterated to us that success is going to mean a lot of different things, depending on the partner we’re working with and their specific goals.
Another point of feedback we heard repeatedly, both from our clients and everyday constituents, was that the video size and placement wasn’t working — the video window was too small, making the text in some videos difficult to read, and it wasn’t aesthetically pleasing since the top of the window didn’t line up with the top of the overview text. We agreed, and worked with the Mass.gov design team to institute a fix. Videos are now placed in a larger window below the overview text — further down the page, but in most cases above the fold, so a user doesn’t have to scroll far to find them.
Next Steps with Video
We’re encouraged by the response we’ve had to video so far, but we’re not done. We’re following up with more: more experimenting, more testing, and producing more videos. We’ve expanded the content types that these videos can be posted on to include how-to pages, service detail pages, and information detail pages, and we’re working with additional agencies to produce new videos to support their content.
We’re always looking for ways to improve our assets and strategies. Let us know in the comments if you have any video tips, tricks, or advice.
If you work for the state of Massachusetts and think your agency has a page on Mass.gov that could benefit from a custom video, you can contact us here.