Earlier this year, NRKbeta, the Norwegian Broadcasting Corporation’s media and technology site, open sourced its comment quiz plugin for WordPress. The site’s publishers have been experimenting with requiring their readers to complete a short, three-question quiz before giving access to the comment form on certain articles. The goal of the plugin was to prevent rants and off-topic responses by ensuring that commenters have read the article.
NRKbeta has published some preliminary results after six months of experimenting with a mandatory quiz before commenting.
“On average, there are a lot more attempts – both correct and wrong – than actual comments,” NRKbeta journalist Ståle Grut said. “It seems many take the quiz to check how much they remember from the story – and not necessarily to leave a comment. Almost as a fun little game after reading.”
Grut reported that on average the quiz has an error rate of 72%. His team suspects that the bulk of the wrong answers are coming site’s international readership, as most of the articles are posted in Norwegian, which can be difficult to translate.
The sample is still relatively small, because the team hasn’t yet created any set rules for when authors should enable the quiz.
“The idea was to test it on stories that had potential for a gloomy comments section,” Grut said. “It is something we are proud to rarely have here at NRKbeta.”
On one story, NRKbeta staff made an error where it was impossible to submit the correct answer to the quiz, because it wasn’t listed. As a result, this article’s quiz received more than 1,000 wrong answers.
One unexpected benefit of the plugin is that it makes it more of a hurdle for readers to leave short comments, such as “nice post” that don’t add much to the conversation.
“This favors the most eager with the most time on their hands,” Grut said. “From time to time this has led to a decline in quality and tone, causing him to often abandon the quiz module.”
These initial conclusions are in line with what we predicted when the plugin was released: the most motivated ranters are not significantly inconvenienced by a short quiz. Keeping comment sections free of trolls is not yet something that is easy to automate. It still requires time spent in the moderation queue.
After the comment quiz plugin was enabled on the site, NRKbeta counted more than 300 articles around the web that had been published about the experiment. Quizzing commenters was hailed as one of the best new ideas for warding off trolls. However, the NRKbeta team cannot yet conclude whether the plugin is a success or not.
“The numbers seem to show that the quiz has worked like a little game for many readers,” Grut said. “They like to take the quiz, but not to leave a comment. Being tested on how much they remember from the article seems to be the most popular use of our quiz.”
A couple monthts ago I try that comment quiz plugin and I have noticed the same. Readers enjoy it but rarely leave a comment.
Even in every single theme I have used there are important differences in quantity and quality of visitors comments.
I guess there is something to be discovered.