User testing

The user test is the most effective method for evaluating usability. It allows to directly observe the user when he is using the application. User testing provides a means to identify the usability problems in real terms. Usability could be measured during the test by calculating the user performance.

Advantages

• The main interest of user testing is to observe the user in a real context of use.
• The problems identified are those that are really raised when the user uses the software.
• The problems are objectively identified by difficulties that restrain the user in his task.
• Measures could be done during the test.

Drawback

• The test could hardly cover all the functionalities of the software.

The user test is an opportunity to see the user in real situation and to observe the problems he raises, the questions he asks and the services he appreciates or not. In this way, the design team can collect invaluable elements in order to make the software easier to use.

Protocol

The user test consists in putting the user in a context as close as possible to the real one.

The observer gives various instructions which will lead the user to achieve standard tasks for which the software or the web site has been developped.

It is essential not to help the user (except, of course, in case of blocking). Actually, in order to identify clearly the issues, it is important to let the user 'manage' as he will do when he will be alone in front of the computer.

The observer notes the errors made, the misundertandings, the blocking points and any event which shows a difficulty of use of the software.

These various observations are analysed with the user, once the test finished, in order to have a better understanding of the reasons of the problem. Innovative solutions are generally raised during these meetings.

During design stage, user testing allows to validate assumptions made during the design about the user behavior, for instance the way he navigates within the interface, the information he seeks or the commands he uses the most frequently.

Usability measurements

In order to estimate the usability in real terms, the user performance is measured. In other words: Could the user correctly achieve the task in the planned time?

More precisely, the ISO 9241-11 standard defines the usability in the following way:

This definition provides objective criteria to evaluate the usability of a software: a software is usable when the user can achieve his task (effectiveness), with a minimum of resources required (efficiency) and that the system is pleasant to use (satisfaction).

Therefore, measuring usability consists in measuring 3 variables:

• Effectiveness: Check whether the objectives of the user are achieved.
• Efficiency: Measure the resources required to achieve these goals, for example the time needed to achieve the task.
• Satisfaction: Determine if the system is pleasant to use, for example by counting negative remarks said by users during the test.

The standard defines the usability on the basis of these 3 characteristics. They are indeed the most representative points in the general case. However, it can be useful, according to the application, to evaluate other aspects:

• Security: Number of errors made by the user and the ease of correction of these errors.
• Ease of learning: Correct understanding of the operating mode and speed of learning.

For each instruction given to the user, measurements are done in order to calculate the above variables. In this way, the project team is able to evaluate the usability in real terms.

Recommendations

During the test, the user does not dare to say that he cannot succeed in using the software or the web site. He prefers to hide his difficulties, distorting the results of the test.

For this reason, it is essential to put him in confidence at the beginning by recalling him that:

The user does not succeed in using the software, because the software has been badly designed. The objective of the test is to identify the usability problems of the application and not to measure the capacity of the user to use the software or the web site.

When the test instruction is well defined, with a precise objective, the test results are more easily usable. Otherwise, the result can be variously interpreted, not facilitating the improvement of the software.

A good adequacy between the test objective, the users involved and the instructions allows for relevant results. For this reason, a very detailed attention must be paid to the specification of the test protocol. A "pre-test" with the team members is generally useful.

The users must be those targeted by the evaluated software or web site. It has to be those who will effectively use the software, otherwise the results are likely not to be relevant.

It happens that some users get used to experiments by attending to many tests. Their attendance show their motivation. They are generally innovative subjects. But are they really representative of the users? Don't their frequent attendances to experiments move them away from the operational use of the software?

For general public applications, like web sites for instance, it is preferable to frequently change the composition of the panel of users.

J. Nielsen has shown that tests achieved with 5 users allow to raise at least 80% of the usability problems. Actually, increasing the number of users does not allow to find more issues. The issues are related to the software, not to the users!

Testing with a greater number of users increase the cost of the test, not the relevance of the results. Rather than achieving a test with 15 users, J. Nielsen considers that it is preferable to make 3 tests with 5 users, and to improve the interface between each test. Of course, when the application targets various types of users, it has to be tested with the various groups

>> More: Nielsen' article: Why You Only Need to Test With 5 Users

© Usabilis - Fontenay aux Roses, Paris - France - Tel. +33 1 46 61 15 31