Going "in the wild"

User research has been a key activity of the AquaBrowserUX and UX2 projects lately. In a previous post, I outlined a scope of user research involving library persona development, usability testing and contextual inquiry. The activities fulfil two general aims, 1) to bolster usability research capability and practice, 2) to evaluate specific usability issues of existing library services. The research involves various qualitative and quantitative methods such that richer case studies can be attained without relying on a single approach. We have just completed the development of library personas (approach and outcomes - two posts to come). Usability testing of AquaBrowser is currently underway.

One of our evaluation goals is to gain insight into how library services are being used in real scenarios involving authentic tasks. This post describes a pilot contextual inquiry, a field study of real users and AquaBrowser usage. The aim is to evaluate the usefulness of the system in a specific use context at the University of Edinburgh.

Contextual Inquiry in the Library

Contextual inquiry addresses some key difficulties of qualitative methods such as survey and interview which require participants to recall answers from memories. It relates to ethnographic study of users, observing them in their natural habitats. This method provides broader understanding of how systems fit into working environments and explores common users behaviours and research areas that are fuzzy and difficult to be delineated and articulated correctly by researchers and users.

We conducted the field study at the University of Edinburgh library (George Square). The study focused on user activities on the library ground floor where AquaBrowser can be accessed through various computer terminals.

   Library Catalogue Terminals in the Main Library

The computer terminals are situated in the main public foyer and a short-loan section further into the building. AquaBrowser is neither the default nor the only library catalogue user interface. It is being presented as "a new way to search the library catalogue", along with the existing 'classic catalogue'. Both are accessible through a welcome screen.

   Welcome Screen

This inquiry is limited to a specific setting: the use of AquaBrowser at public library catalogue terminals. There are other ways of accessing the UI in the library. For example students can search the catalogue using personal or library computers at the study facilities such as open-access computing facilities, group and individual study spaces. Mobile devices are also becoming popular in the campus and may also be used as an alternative means to access the catalogue. All these entail different use contexts, research challenges and therefore merit new studies beyond the confine of the projects and certainly the scope of this post.

Approach and Scope

The study was conducted over a course of 1.5 days (3 half-day observation periods) in May during the university examination period. The approach:

• Survey the library ground floor (main foyer and short-loan section) for AquaBrowser usage
• Approach AquaBrowser user at terminals
• State intention: brief one-sentence introduction
• Seek permission to observe and to have post-session interview
• Observe session
• Conduct in situ interview (if the user is free): brief follow-up and general questions

The observation is informed by the UX2 user research scope and focuses on:

• User goals: purposes of use, what are the 'stop conditions'?
• User behaviours: what is the pattern of use, how users access the service
• User attitudes: how the users feel about the service or perception of UI features

Caveat: only a small number of AquaBrowser sessions were observed during the first observation period. Due to time and project constraints, we had to bypass the welcome screen and set AquaBrowser as the default UI at the terminals for the subsequent observation periods in order to gather sufficient data for the inquiry.

Classic UI and AquaBrowser - first observation period

Given the options presented at the welcome screen (classic catalogue UI or AquaBrowser as "the new way to search"), the first observation is about the user's choice between the options. During the first observation session,

• 25 sessions were observed
• 23 users chose classic catalogue UI, 2 chose AquaBrowser

Most sessions lasting only several minutes. There was one long session lasting around 20 minutes by a user who seldom visit the library and prefer to search the catalogue from the office. Since the usage of classic catalogue UI is not within the project scope, we asked the users where possible about their selection. Below are some of the reasons cited for choosing the classic UI:

"Simple, available through MyEd (and WebCT)"

"Just hit the big blue box [link to classic UI], didn’t border to read the links"

"Not aware of AquaBrowser" (4 users inc. 1st year and 2nd year students gave similar remark)

The lack of awareness for AquaBrowser seemed to be common, although it is difficult to attribute this to the choice selection without further investigation. Perhaps users are 'UI agnostic' at public terminals and pick whichever UI they come across to get things done. The first two remarks above allude to this. The blue button in the welcome screen actually encourages the use of classic UI. This is an example of Fitts Law in HCI (size and selection times). A more neutral design of the welcome screen may encourage AquaBrowser usage, e.g. adding an AquaBrowser button next to the existing one or just employ two larger and similar size links.

Below are the remarks from the two AquaBrowser users regarding their selection:

"Simple and clear. Good design."

"I hate the classic search, would actually switch to AquaBrowser if classic was on the screen. Classis is too specific and quite limited. You need to know exactly what you are looking for."

Simple design strikes again, and for both UIs too. The fuzzy search capability of AquaBrowser appears to be a winning feature for one of the users.

AquaBrowser Usage - all observation periods

In total, 17 AquaBrowser sessions were observed. The following are the commonest goals:

• Resource discovery: searching for books and holding details, looking up books from a reading list, 15 sessions
• Looking up shelf-mark (for the terminals in the short-loan section), 2 sessions

It is interesting to note that the proximity of bookshelves to the terminals (in the short-loan area) has an influence of AquaBrowser usage. Some users only used the system to look up the shelf-marks of nearby bookshelves and prefer to browse books physically. Perhaps there is a case for custom and simpler UI dedicated to shelf-mark lookup? It might work for mobile devices. Most users (15) fulfilled their goals. The main stop conditions are:

• Found resources ("found the book(s)", "found the shelf-mark")
• Found enough resources ("have enough to go on for now")

Two users however didn't complete the search tasks:

One abandon task after a single search because she cannot remember the book title and decided to look it up on the internet first (the terminal has no internet access).

One couldn't find the book and decided to ask a nearby librarian for help.

There is another notable stop condition. Confronted by a used terminal (with search term/results left from a previous session), almost half of the users (8 out of 17) attempted to select the ‘Library Online’ or ‘Classic Catalogue’ links in the AquaBrowser UI - see below. This action is prompted by a desire to create a new/fresh session (empty search box and front page), as confirmed by some users during the post-session interview. One user did this at the end of a session in order to leave an empty page for the next user.

However, the links do not work because the terminals are not internet-enabled and result in error messages at the time of writing. Instead of replacing the existing keywords in the search box and initiate a new search, some users abandon terminals and moved to other terminals in search of new sessions. This results in a terminal-hopping behaviour. In normal circumstances (when AquaBrowser is not the default UI) the user is likely to end up at a terminal with the Classic UI which has a 'start again' feature.

Currently, the only way to initiate a brand new session for AquaBrowser is through the welcome screen where user can opt for the UI again. The welcome screen can only be activated by a 'CTRL-W' keyboard entry. But this instruction is neither obvious nor clearly labelled at most of the terminals. To avoid these context-dependent idiosyncrasies, AquaBrowser needs a straightforward 'start again/clear' function especially for public terminals use scenarios. The observations have shown that users rather hop around and end up with the other UI, than replace existing terms of other users in the search box.

The main activities observed:

• Keyword search
• Review results
• Make note on papers

Most users repeated the above steps in the same session. Some made minor deviations, for example reading table of content, browse more books using the shelf-mark links and faceting (see below). A majority of users scribbled notes on paper repeatedly. This perhaps implies a need for extended services such as annotation and online bookmarking. AquaBrowser provides additional Web 2.0 type of services through My Discoveries. At the time of writing, My Discoveries has not been piloted by the university. Otherwise, it would make an interesting case for usefulness and usability testing.

The information seeking behaviour can be studied through the intricacy of the above activities and the deviations, how they vary and iterate over time. Marti Hearst (2009) provided a good summary based on the work of Bates (1979) which characterises searching as straightforward (look up one particular book) or a strategic process involving various type of tactics. The following is a breakdown of the types of information seeking behaviour observed:

Straightforward (keyword search once -> result) (7 sessions)

• looking up a book (4)
• looking up a shelf-mark (2)
• repeatedly looking up known books from a reading list (1) (typing long book titles in the search box!)

Tactical (6 sessions)

• term tactics (4): repeatedly modifying search terms based on results reviews or following suggested terms provided by the system. Repeat frequency varies from once to 4 iterations at most.

• information structure tactics (1): immediately navigate to a link within a known information structure to obtain results.

  The user browsed and noted a list of books via the shelf-mark link of a book details page obtained from a straight-forward search. This is analogous to physically going to a bookshelf with shelf-mark of a known book and browsing other books there. There are other metadata links in the details page e.g. subject headings and author, plausible for use in this tactic. However, they were not used.

• query reformulation tactics (1): immediately narrowing a search through Boolean operations.

  The user undertook keyword search of a subject (once) and subsequently obtain results by navigating around the faceting (refine) facilities on the right - see "Faceting and Word Cloud" below for more details. The user repeated this process.

Strategic (4 sessions)

The above tactics used in combinant.

• term tactics + information structure tactics (2): both users deviated from keyword search sessions (term tactic) and made use of the shelf-mark links to find more books.

• term tactics + query reformulation tactics (2): both cases ended keyword search sessions with the use of faceting to refine results. One user selected an author facet once, the other made use of a subject facet and a publication date facet (last 5 years).

Overall, the usage is dominated by straightforward and single-tactic information seeking behaviour. This is not surprising given the use context.

Faceting and Word Cloud Usage

Faceted search provides a “navigational metaphor” for boolean search operations and enables fast drill-down of results. The facet links are essentially advanced search shortcuts. Selecting a link after a search is equivalent to performing a query reformulation tactic and a post-coordinate boolean operation. Out of the 17 sessions observed, 3 users made use of faceted search to accomplish their tasks successfully. This alludes to the usefulness of this feature which is becoming common in digital libraries. A user who did not use faceted search, noted that it is a feature he has used, mostly to refine results by author.

In one particular case, faceted search solely underpins the user's information seeking behaviour. The user began with a search of the subject ('Scottish Romanticism') and immediately selected the 'more..' option in the subject facet to open up a big subject headings list (see below) from which a facet ('history') was selected - see below. This led to a book being found. She repeated the process again to find another book. Follow-up question revealed a positive attitude towards the faceted search feature: "well ordered, great for finding stuff quickly".

   Subject Faceting 1

         Subject Faceting 2

The AquaBrowser homepage by default does not feature any facet listing. Faceted search is only a post-search feature for refining results. The user (from the above case) seems to use keyword search as a means to activate and access the facet listings. If a relevant facet listing is available in the homepage, perhaps she would forego the search and navigate her way entirely, from facets to facets to find books.

An interaction design pattern we explored earlier this year, through a UX2.0 heuristic inspection, is the unique "word cloud" feature which provides context-sensitive exploration and serendipitous discovery in AquaBrowser. We did not observe any word cloud usage during the observation periods. During the post-session Q&As, one user mentioned about occasional usage, while another expressed liking this particular feature as it provides very useful information and details. It remains to be seen if word cloud will find usage in this particular use context (public terminals), and indeed at other library locations where research activities and serendipitous discovery tend to occur.

There is however one particular usability issue we observed that is related to the word cloud. 3 users collapsed the UI (panel) by clicking on the arrow-like button - see below. These users confirmed during the post-session interview that they mistook the button as a back button. None of them re-open the UI and continued with their search tasks.

Note

This post only describes a brief pilot study highlighting a specific use context of AquaBrowser. It is by no means exhaustive and representative of other use scenarios, for example usage at open-access computing facilities and study spaces with personal computers. More information of AquaBrowser usage is forthcoming through the user research conducted for persona development and the usability testing.

Thanks!

Comments

[Helen Harrop]

Looks like you've uncovered some really useful insights around user search behaviour here - There's so much information it's hard to know where to start but the things that stood out to me as particularly interesting were:
The users' reluctance to do a search unless it was from a 'clean slate' ... it almost sounds like there's a case for there being a timeout function which returns the terminal to the welcome screen (although that could be fraught with the risk of making the user experience worse).

I noticed that the user who searched for Scottish Romanticism makes a typo in their search term (romantism) but the search results gave them what they were looking for which is always good from a UX point of view.

Did you get any feedback on the appearance of the word cloud? The example in the screenshot looks (to my eyes) more akin to a category list than a typical web 2.0 style word cloud so it would be interesting to know the users' perspective on that.

I look forward to hearing more about it in Glasgow. Have a lovely weekend!

Helen (Project Synthesis Liaison, JISClms)

[Boonious]

A timeout function could be the next best solution, although some survey/measurement should be done on the timeout duration. We didn't video or capture screen. I just reproduced the screenshots based on the notes taken on the user's steps. So the typo was mine! And I'm surprised too about the results. I guess this kind of validate one of the user comments about the fuzzy search. But the same can't be said about the spelling test we are conducting in the usability test right now. Yup we have some feedback on world cloud both from the user research and usability testing. If I have time in Glasgow I will show you some videos.