Fact is that you can often learn more about how usable a product or service is by observing what users do than by listening to them. There is usually a big difference between what a user say she would do and what she would actually do. In the end, it is what the user does that counts.
This week the results from two different usability studies have been presented. The first study was a qualitative test with users where a limited number of users were given some pre-defined tasks to perform on a prototype. They were observed then and interviewed about their experience. The conclusions were primarily based on the results from the interviews.
The other one was an analysis of usage statistics from the live site. It used a combination various technologies to measure the performance of the existing site in terms of usability.
My particular interest was in the findings that related to the search engine. One such finding from the qualitative user tests was that only one out of eight users said that they would use the search function as their first choice when given the task to find a specific type of product. Seven of them said that they would use the taxonomy-based navigation and browse to find the product they would look for.
Looking at the results from the quantitative study, it gave a completely different picture. One finding was that more than 60% of the users visiting the site today chose to use the search and only around 10% went browsing through the taxonomy-based navigation. The rest chose other alternatives, probably because they were looking for some other information than products.
Given that the search function will be more prominently located and easier to spot in the new design (which the prototype in the qualitative tests were based upon), one should not draw to many conclusions from the results from the tests with users. Just the fact that the users were participating in a test is likely to have affected their behaviors. The interview probably also made them think to much and analyze their own behaviors during the test. On a live site, users just do things to complete the task they are there to perform. They act on intuition and habit, choosing the path that seems as the most natural way to satisfy their needs.
I expect 70% or more users will use search as their primary entry point when looking for products and other information on the site after my project has launched. Given that the search is the fastest way to find a product - the navigation always require several clicks while the search potentially only requires two - I am convinced that the changes we are doing will also lead to increased sales. The real data on real user behaviors that we will capture and analyze before and some time after the release of the new search will then show if I am wrong or right.