Gutenberg Team Is Ramping Up Usability Testing at WordCamp US

The Gutenberg Team will have a usability testing station set up at WordCamp US where attendees can participate in a round of pre-set tests that focus on the writing flow. Testers will answer a short survey that includes their prior WordPress experience level, age, and device used. Volunteers will get participants set up with a testing site and will start the screen recording app.

Testers will be asked to create a post based on the content shown in an image. There are three different images, which require the user to perform actions such as adding images, embedding media, creating unordered lists, adding quotes, and other basic content creation tasks. In order to segment results, the usability tests have been divided into beginner, intermediate, and advanced level images.

Advanced level task image for Gutenberg usability testing

After completing the test, participants will be asked to answer a few followup questions, such as “Did the task take longer or shorter than you expected?” and “Are you more or less likely to use the Gutenberg editor in the future?”

“This is the second round of usability testing scripts — we tried out the first batch of scripts at WordCamp Milano, and made some adjustments for clarity,” Gutenberg design lead Tammie Lister said. “As a result of testing, we moved the toolbar on blocks to not be fixed and back to the block. At Milano, we tested the tests.”

As the result of these tests and other prior feedback, Lister recommended the default position of the toolbar to be fixed to the block.

Anna Harrison, UX lead at Ephox (the makers of tinyMCE), has been instrumental in helping with the efforts around testing and writing scripts. She also offered feedback on the ticket, referencing comments from the previous discussion on the issue:

A fixed [docked to top] toolbar solution has several complications. Firstly, we break accessibility. I won’t reiterate the discussion, as it’s well articulated above. Secondly, we break things independent of accessibility – I ran user tests on something quite similar to this last year, and we discovered that disconnecting the toolbar from the point of action resulted in 100% user test fails.

Gutenberg version 1.8 will change the default back to displaying block actions on the block level, although the option to change it to a fixed toolbar at the top of the screen will still be available. This change is one example of how usability testing is shaping Gutenberg’s development. WordCamp US is an opportunity for the team to collect a host of new testing data in one place.

Lister said all the data that is collected will be processed by volunteers on the make/test team, but the team is still small and they could use more volunteers to work on this effort.

“The turnaround time on processing the data we collect really depends on how many volunteers are available to work on it,” Lister said. “It also depends on if it’s a bug reported – bugs are easier to get fixed right away. If the data indicates an area where we need to investigate more, we’ll do that. The results of the testing will be published on make.wordpress.org/test.”

Lister said the team is hoping to reach a wider variety of WordPress users at WCUS this year, from all backgrounds and careers. The testing booth offers an opportunity for anyone to contribute to the future of WordPress, regardless of your experience level or familiarity with the software. The team is also eager to broaden its testing field by recruiting non-WordPress users as well. If you can’t make it to WordCamp US, you can still contribute to Gutenberg by taking and administering usability tests on your own with the help of the instructions posted on the make.wordpress.org/test site.