Institutionalising Human Factors in the Design Process: The ADONIS Experience
Andrew Dillon, John Richardson and Cliff McKnight
This item is not the definitive copy. Please use the following citation when referencing this material: Dillon, A., Richardson, J. and McKnight, C. (1991) Institutionalizing Human Factors in the design process: the ADONIS Experience. Contemporary Ergonomics '91, London: Taylor and Francis, 421-426.
The ADONIS workstation was an experimental document delivery system developed to facilitate retrieval and printing of CD-ROM stored articles. Although primarily aimed at the document supply industry, its potential application in libraries and research environments with end users was recognised. Hence, the present authors were asked to assess the system ergonomically from this perspective, and the outcomes of this assessment form the basis of this paper.
It was clear that the design of the system ignored many formal ergonomic considerations. Subsequent user evaluations of the interface highlighted several potentially serious problems and cast doubt on the usability of the system in the end-user domain. A follow-up study of actual usage rates during field-testing of the product confirmed the findings of the initial evaluation and also indicated that some of the very first design decisions (e.g., regarding the choice of journal titles included), which proved detrimental to the system's acceptance, might have been very different if human factors considerations had been taken into account.
The present paper describes the system and the problems highlighted by the human factors practitioners as well as discussing the pitfalls that could have been avoided by earlier involvement. The potential role of human factors in early design of such products is emphasised.
Damodaran (1991) contrasts the ideal role of the human factors practitioner with that typically experienced. The goal of every practitioner should be to be involved from a project's inception as one of the design team so that the first, and often fundamental, design decisions can be fully informed with regard to human factors. Only then can human factors be said to have been institutionalised in the design process. Unfortunately, all too often the human factors advisor is requested to provide 'palliative ergonomics' to remedy design faults. The following casestudy highlights this mismatch between reality and the ideal state.
The ADONIS Document Delivery workstation was designed to facilitate retrieval, viewing and printing of CD-ROM stored articles. The system was developed in order to investigate the extent to which information technology could reduce the costs of labour intensive photocopying procedures in the document supply domain and to increase copyright control over published material (Campbell and Stern, 1987). Biomedical journals were selected for the trial service on the basis of journal usage studies (Clarke, 1981). Throughout the life of the ADONIS project, a total of 219 biomedical journals were optically scanned and the resulting bit-mapped images stored on CD-ROM. A new CD-ROM was issued approximately weekly.
The workstation consisted of an IBM PC/AT or compatible, a CD-ROM drive, a high resolution (300 dots per inch) A4-size monitor and a laser printer. Basic interaction with the system required the user to enter data via a form filling screen and select options from a menu of commands. Loading of CD-ROMs was also necessary for the display of full text and printing. Thus interaction was constrained to a relatively simple sequence of tasks.
Since the articles were scanned at 300 dots per inch, the screen display was substantially the same as the laser printed output. The display software allowed simple page-turning in either direction, but only a single page at a time with no jumping to either beginning or end of the article. The fact that the articles were stored as bit-maps meant that only the separately prepared author/title index could be searched.
Intended users of the ADONIS system were document supply centre personnel who came to the workstation with specific article requests. Since they employed the technology as part of their normal work duties they will be referred to here as the dedicated users. However, the system also had potential as an in-house library system for scientists and scholars, who could search for and retrieve articles for personal use. They will be referred to here as casual users. The authors were asked to evaluate the system from both perspectives although the present paper concentrates on the potential role of the system for casual users.
2. The Evaluation
The full evaluation involved two studies. In the first, an evaluation was made of the user interface, both for dedicated and casual users. In the second study, a workstation was sited in a medical library and made available to medical staff (casual users) whose usage was monitored.
2.1. The Interface Study
This study involved an interview and walkthrough procedure with four dedicated document supply centre users on-site and a lab-based usability evaluation with ten casual users. The ADONIS interface consisted of a form filling screen for specifying search parameters. Interaction beyond this point involved selection via menus of a retrieved article and manipulation by keypress of the displayed text. Thus the user interacted with a limited range of displays in the course of normal task performance. The following sections briefly summarise the main findings from these evaluations.
2.1.1. Form Filling
In terms of the human factors literature, form filling and menu selection have been well researched as suitable means of interaction, and guidelines for successful design have been proposed. For example, Shneiderman (1987) lists nine guidelines for the design of form-filling interfaces, and these represent a good condensation of the findings of human factors research in the area. The ADONIS interface can be evaluated against such guidelines, both for dedicated and casual users, and we have reported the results of such an analysis in full elsewhere (Dillon, 1988). In brief, the interface was found lacking in several areas. For example, grouping and sequencing of fields was not logical for either user; although dedicated users entered the information in a way that 'worked around' the interface, casual users found the sequencing of fields far from intuitive in terms of their normal method of remembering bibliographic details. Help facilities, though presented as an option, were either extremely limited or not available and this caused problems for some users.
Generally, ADONIS was perceived as providing little explicit feedback. Particularly commented upon was the feedback provided when a search is being carried out. The user is informed only that the system is "searching" and provided with an option to interrupt this activity. However search times of up to seven minutes have been noted and such feedback is insufficient for users waiting on a response. The dedicated users pointed out that when batch printing, the system fails to discriminate between signalling that a print out has been completed and that a new CD-ROM is required. Both result in a single "beep" from the computer. Thus if working elsewhere while the system is batch printing one must guess which signal is which and regularly check the status of the machine. This removes a major advantage of the system, i.e., allowing users to concentrate on other work while articles are being printed.
2.1.3. System Messages
The language of the system messages, prompts and so forth, is another crucial human factors issue. In ADONIS, error messages exist which are extremely hostile and uninformative. For example, if the user selects the Report generation option from the top menu when the printer is not switched on, the system responds:
Fatal error in <report_main> code 0001 ffff
From a human factors perspective such messages belong to the 'dark ages' of computer use, offering no indication why an error occurred or what a user should do to rectify the situation - essential features of any diagnostic message.
2.1.4. Input Formality
Input formality refers to the extent to which user input must satisfy specific and rigid parameters for successful interaction. Although ADONIS allows abbreviation of search criteria, it is extremely intolerant of simple inclusions or omissions of punctuation. For example, the presence of a comma between an author's surname and initial when searching on a name only has been found to produce results different from an identical search without the comma. Similar effects have been noted for the presence or absence of a hyphen in article titles. Simple spelling mistakes have the same consequences. This is extremely irritating and can lead users to draw totally incorrect conclusions from the database.
2.1.5. Text Manipulation
As was stated earlier, users can only manipulate displayed articles by paging forwards or backwards, one page at a time. There are no facilities allowing the user to jump directly to points in the text, not even the beginning or end of the document. ADONIS stores documents as bit-mapped images rather than searchable ASCII files, so manipulation is curtailed. From a user's perspective, this is a major disadvantage to using the system.
2.1.6. General Ratings
The casual users were highly critical of the system. Apart from the issues described above, only two subjects said they would consider reading an article with such a system. Of the rest, two said they would never use the system and six said they would consider using it for retrieval purposes and selection of relevant articles.
In summary, therefore, the interface study suggested that casual users in particular would experience difficulties in using the system, and that significant design improvements could be made even for dedicated users. On the basis of these findings, coupled with the existing knowledge of the problems of electronic text usage in general it was predicted that the system as it stood would fail in the casual user context. These findings were made known to the product's developers who felt that in general they were "favourable" and that the system had in effect "passed" its human factors evaluation.
In order to test the system in relation to its possible function as a (partial) scholar's workstation, it was then decided to make the system available to a relevant end-user population. However, since only a single system was available, it was not possible to provide systems for individual users and it was therefore decided to site the system in a suitable library.
2.2. The Library User Study
2.2.1. The Approach
A two stage approach to evaluating the ADONIS system in the library context was identified and adopted. The first stage consisted of allowing relatively unrestricted access to as many users as possible (with no financial charge) and closely monitoring the use they made of the system, with specific reference to any problems that were encountered. Consequently a non-directed observational approach was adopted. By allowing users to determine their own degree of usage a measurement of the system's usefulness could be made. The second phase consisted of a series of interviews and questionnaires to measure the users' impressions of the system and the value they attached to the facilities it provided.
To facilitate the observational approach, several modifications were made to the software in order to log user activity automatically and write the usage data to a logfile. Reports based on the database searches were printed and stored for later analysis. We have presented the results of analyses more fully elsewhere (Richardson, 1988).
The anticipated level of usage (as assessed by the medical library staff) was far higher than that actually obtained and the intended follow up survey was reduced accordingly. Some 90 people were identified at the beginning of the trial as being likely to have an interest in the ADONIS material on the basis of previous inter-library loan requests and the knowledge of the library staff. Although these potential users were contacted directly and informed about the system, only 24 responded and 22 decided to attend a demonstration. During the course of the trial another 25 people expressed an interest and received a demonstration. However only 13 of those who attended a demonstration then went on to make use of the system. It was decided that all the users should be surveyed and that a sample of 20 of those who had not responded at all or who had only attended a demonstration should be contacted and their reasons for not using the system investigated.
Despite the low level of usage, a number of conclusions can be drawn that have implications for end-user implementations of the ADONIS system in particular and for the design of future electronic journal systems in general.
The major determinant of the system's usage was the range of journal titles covered and this has a bearing on both the specific and general application area. The bio-medical field was chosen because of the high demand for inter-library loans. However, this level is observed only at the national level and, to a much lesser extent, at the institutional level. Although ADONIS covered 219 journals this is relatively small compared to the 2500 titles included in, say, Medline. In addition some of the area's major titles were missing from ADONIS since only journals published by the ADONIS consortium members were contained on the CD-ROMs.
The individual bio-medical user has a limited demand for material in a very narrow sub-area. If the ADONIS system does not have a comprehensive coverage of an individual's specialist area then its utility will not be highly regarded. The end user is likely to consider a bibliographic database like Index Medicus much more useful if subsequent delivery of the full text is reasonably swift. A thorough search of these databases is unlikely to result in important references being overlooked. The user who starts a literature search using ADONIS is likely to need to follow it by using the other bibliographic databases as well. Hence, to the individual user, the ADONIS database is simultaneously too broad because it covers such a wide range, and too narrow because it omits several important sources. This is an issue that could have been captured in a user requirements specification at the earliest stage of design.
Interestingly, despite the results of the earlier interface evaluation, the users in this trial generally regarded the system as useful and not difficult to use. However their actual use of the system was limited to fairly simple search strategies. While this could have been a response to the unfriendly interface and the constrained data format, it also suggests that user training regarding effective search strategies is important.
The observed tendency to use the screen display option to read abstracts only is difficult to interpret. Although the quality of the screen display was not mentioned as a reason for not reading articles, it was quite clear that the laser printed versions offered a permanent, personal copy. The constraints imposed by the Adonis system design prevent a serious consideration of the possible advantage of an electronic journal system over the current paper based system. To investigate this issue fully it would be necessary to carry out user trials with users gaining access to a workstation on their own desk to mimic the ease of access that they currently enjoy to their personal filing cabinets. In addition, the electronic database would need to be searchable (i.e., not in facsimile format) and also provided with an interface that was specifically designed to meet their needs.
The library usage study confirmed the prediction of the earlier user interface valuation, albeit for different reasons. The system was not a success with end-users in libraries and would require modifications from first principles to improve.
3. Being right but being ignored
The heading of this section is not intended to suggest that human factors has all the answers or that the findings of the evaluations were ignored by the developers, they were not. There is no doubt that our conclusions were justified and correct. Human factors does have a role in systems design, the tragedy is that it is seen as such too late, i.e., it is ignored for large parts of the design process. Had some attention been paid to the human factors issues in the early stages of design rather than at the point of delivery where only evaluation of an instantiated design was possible, it is estimated that real improvements could have been made. It was clear to the evaluators after only a preliminary examination of the system that there were problems. Subsequent evaluations confirmed this diagnosis.
It is not difficult to identify the potential role of human factors experts in such a design process. For example, at the specification stage, information from intended users could have been obtained by observation and interview rather than relying, as the consortium did, on usage statistics from libraries. This would surely have led to a more enlightened selection of material. Initial designs of the menus and form-filling screens could have been examined by human factors professionals who could have easily identified many of the problems reported in section 2.1. Prototypes could also have been evaluated. Any or all of these activities would have changed the design, in our view, for the better.
The area of electronic text design has been subject to much investigation in recent years, not least because of the emergence of hypertext (gratuitous reference number 1!) Current knowledge still contains many gaps but there is an emerging body of research findings that support some theoretical perspectives on reading from screens that would be of practical use to system designers in this domain. The goal of ergonomics now must be as much to apply this knowledge as create it so that we may be right and be listened to.
Campbell, R. and Stern, B. (1987) ADONIS - A new approach to document delivery. Microcomputers for Information Management, June.
Clarke, A. (1981) The use of serials at the British Library Lending Division. Interlending Review,9, 111 - 117.
Damodaran, L. (1991) Towards a Human Factors Strategy for Information Technology Systems. In B. Shackel and S. Richardson (eds.) Human Factors for Informatics Usability. Cambridge: Cambridge University Press.
Dillon, A. (1988) The ADONIS document delivery workstation: a human factors evaluation. Project Quartet Deliverable PQ/LUT/26, HUSAT Research Institute, Loughborough University.
Richardson, J. (1988) The ADONIS document delivery system: a library trial with end-users. Project Quartet Deliverable PQ/LUT/28, HUSAT Research Institute, Loughborough University.
Shneiderman, B. (1987) Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA: Addison-Wesley.