TIME: A multi-levelled framework for evaluating and designing digtial libraries
This item is not the definitive copy. Please use the following citation when referencing this material: Dillon, A. (1999) TIME - A multi-level framework for the design and evaluation of digital libraries. International Journal of Digital Libraries 2(2/3), 170-177.
Digital libraries promise benefits for all stakeholders in the information usage community which almost certainly will be matched by commensurate shortcomings that are as yet unforeseen. Even though these are very early days for digital libraries, three decades of research on human-computer interaction in general, and the recent history of hypermedia research in particular, indicate that whatever else occurs, the usability of digital library applications will prove crucial to widespread acceptance. In the present paper an evaluation framework (termed TIME) is outlined. TIME offers designers and implementers of digital libraries a framework to address key human factors in a user-centered manner. Bridging all levels of human factors, from the ergomonic to the user goal, TIME is a scoio-cognitive framework that highlights the interplay of multiple issues affecting user response to digital documents.
Since the earliest studies of user interface design, the field of human-computer interaction (HCI) has produced numerous experimental findings on the human response to information technology. Grounded in the scientific tradition of empiricism, HCI has frequently cast doubt on intuitive or common-sense ideas for interface design. For example, Norman (1991) showed that contrary to the invocation to limit menus to 7+/-2 items, breadth is usually better than depth in terms of target location. Gould et al (1987) showed that image quality counts for more of the reading speed deficit between screens and paper than some of the more 'obvious' variables of screen angle or user experience, etc.. To this we might add that zooming effects for opening and closing windows are not advantageous to users who are not experienced in their application (Chui and Dillon, 1997), user cognition is not simply reducible to the number of steps involved in a task (Gray et al, 1993), or that even when designing for users who are like themselves, designers cannot reliably predict how well these users will perform with a system (Bailey, 1995).
There are two important lessons for us to learn here: interfaces matter; and usable designs are not the result of intuition on the part of the designer. Wonderful functionality (e.g., desktop delivery, archive access, linked discussions etc.) is only of value if the user can exploit it, and this is entirely contingent on the interface provided. How we design the interface matters enormously. To ensure good design, we cannot rely on the impressions or common-sense reasoning of designers, no matter how able they are. Predicting others' reactions to a technology is demonstrably difficult, compounded by the difficulty users experience in estimating their own reactions before using a technology in real contexts.
Focusing specifically on hypermedia (Dillon, 1994), I previously outlined empirical evidence indicating differences in both outcome and process terms. Specifically, there exist five basic outcome differences:
and two major process differences:
The human factors of interface design for information usage can best be thought of in terms of physical, perceptual, cognitive and social factors, each representing a particular granularity of analysis and a set of issues embedded in an interwoven temporal sequence. By this I mean that it is possible to address each level as a distinct research issue (i.e., to study perceptual phenomena distinct from social phenomena). Certainly this is the way HCI research has progressed. However we must also be aware of the coupling of all levels in a user - i.e., any user exists at each of these levels in one organism, and does not herself make such distinctions in a fixed manner. Hence our research paradigms in HCI need to recognise the big picture and not assume explanations at one level (e.g. the physical or the social) will either solve our problems or be completely unrelated to issues at the perceptual or the cognitive levels.
To date, we have made most progress in understanding the physical and perceptual factors affecting usability. Gould et al (1987) demonstrated that image polarity, screen resolution and anti-aliasing in combination significantly affected reading speed, and that any interface that failed to address these low-level issues would serve to slow readers of digital documents by as much as 30%. Similarly, findings on optimum input devices (Card et al, 1983) showed the mouse to be maximally efficient with respect to human information processing for pointing at objects on screens. While such clear demonstrations of the strengths of key interface variables are important applications of theoretical models to practical concerns, they are primarily issues that impact the preconscious processes of the user (though of course they may manifest themselves in very conscious reactions to the technology).
Such findings, which I will collectively term here as the 'image quality principle', provide guidance on the minimal standards of acceptance for digital documents . It is my experience that even these are too little known among designers, and digital environments continue to be created that violate the image quality principle, with the resultant predictable reaction by users. One has only to examine many web pages to see how little understood is the impact of image quality. In the framework outlined in this paper, the physical and perceptual issues are included, but they are only part of the story of good design and will almost never be sufficient for ensuring a good interface design that will encourage use and acceptance of digital documents.
As well as image quality, attention must be placed on crucial issues related to user cognition and social interaction. In digital library scenarios we are envisaging users navigating and trying to comprehend large information spaces involving mutiple document types and forms. Document forms represent the conventions of discourse among communities of practice, giving rise to expectations of form and protocol. Part of the problem for most hypermedia applications to date has been the failure of many designers to take account of such variables in their designs. No wonder then that hypermedia technolgogy has failed to produce demonstrable improvements in learning scenarios (Dillon and Gabbard, 1998).
Part of the problem for HCI at this level is capturing and measuring variables that serve as indices of these cognitive and social forces. It is clear that there are major navigation problems with linked digital information spaces but their resolution is not easily obtained through a narrow focus on screen design. I have invoked the concept of information shape elsewhere (Dillon, in press) to describe the mixing of spatial and semantic properties of information that impact users but there are others. As applications become more networked and information access increases to incorporate larger multimedia formats as befits a digital library, then the interplay of spatial and semantic forms becomes more crucial ot users. And since semantics reflect processes involving meaning construction and discourse, the questions to be answered extend beyond physical ergonomics or image quality variables towards questions concerning cognitive and social processes.
Incorporating HCI into the design process
The findings to date offer yardsticks against which we may measure our interface designs, but there exist a significant number of parameters that we cannot fix on the basis of experimental findings to date. Indeed, there is an argument that no amount of research into interface variables will ever yield a sufficiently powerful science-base for design. Consequently, we need to employ a process for design that enables repeated checks of appropriateness in the contexts the technology will find itself. Thus, the advent of user-centered design processes (and I use the plural deliberately, as there is no single agreed process) have evolved to allow us to proceed with design even where we lack findings or evidence on what features we should be employing.
User centered design processes emphasize design-test iterations, the testing of candidate solutions with representative users, and to a certain extent, participation of users in the decisions on designs. Frequent use is made of user and tasks analyses, rapid prototyping and evolutionary change as the design team seek to base design choices on data from intended users. As a methodology, user-centered design processes reflect a strong social science underpinning in their use of test methods and data collection. While the precise form varies, most user-centered processes seek to measure usability with real users rather than assume compliance with style standards or findings is sufficient. And this is crucial to the process I advocate. The design team must make explicit the user, task and environmental variables impacting a technology's exploitation in both the design and testing of the artifact, for without these being articulated, the evaluation of usability is largely meaningless.
The failure to adequately support the measurement of usability can be seen historically in the use of this term in the literature. Usability has been treated as a fuzzy concept, usually defined in one of three ways:
The first type of definition has little utility for design purposes since it merely restates the issue, using equivalent words. What has proved more durable is the feature-based definition. With the advent of design guidelines and style compliance handbooks, people have searched for the optimum form of interface design and sought to express the rules in terms of feature provision. This approach rests on an assumption that usability is an inherent part of the application. In my view this assumption is false. We can demonstrate this quite easily by a thought experiment. If one developed an application that conformed to any idealized style guide it would not matter how elegant or how in conformance with usability principles it appeared, one could always envisage a combination of users, with certain task demands, in a particular environment, for whom the design would be sub-optimal. As soon as one realizes this, then it is clear that the operational definition is the only way forward.
Shackel (1991) is the major developer of the operational approach to usability. He defined usability even in the early 1980s as the artifact's capability, in human functional terms, to be used easily, effectively and satisfactorily by specific users, performing specific tasks, in specific environments. From this definition it is but a short step to the ISO 9241 (part 11) draft standard on usability. The operational approach explicitly places usability at the level of the interaction between users, performing tasks in certain environments, and the artifact. Accordingly, any one interface might prove both highly usable in certain contexts, and highly unusable in others. This takes it beyond the typical features-based definitions common in the field. Second, it operationalises, and thereby provides us with a means of evaluating, an artifact's usability.
We may thus define usability in operational terms as the the degree to which certain users, performing certain tasks in specific environments, can effectively, efficiently and satisfactorily employ the techology to achieve their goals.
Before we reject the feature-based approach however, it is worth reconsidering some of its virtues, even if this complicates the picture somewhat. While we cannot prescribe, a priori, interface features that will guarantee usability (by defining it into the product, so to speak), it is true that certain features are more likely to enhance usability than others. Thus, and there is solid empirical research to support these examples, broad menus are generally better than deep menus, black text on white is better than white on black, natural language error messages are better than coded numerical ones etc. Hence, we can at least point to features that enhance usability, the argument goes. So perhaps a strong line of research will eventually yield the formula for prescribing usability in the a priori manner one seeks.
Seductive as this argument is, and it is at least based on the truth that certain featural combinations do affect the quality of interaction users have with an interface, it is based on a flawed premise. Usability is not a feature, nor a combination of features, but a context-dependent measure of interaction. Features vary in terms of compatibility, and while we may one day (at least theoretically) map out the full range of features that are maximally compatible with the human physical, perceptual, cognitive, and social systems, at the level of real-world work, such a compatibility would never be sufficient to guarantee usability (as defined either operationally or semantically for that matter) since the task determinacy of tool use renders usability contingent on factors that qualify any simple compatibility match.
The operational definition of usability is most closely identified with the approach to design known as usability engineering (Nielsen, 1993) which advocates specification and early testing of designs for usability. Usability engineering itself sits within a broad framework known as user-centered system design, a philosophy of design that emphasizes early and continuous focusing on users, iterative testing and re-design. While a user-centered approach makes the development of usable and acceptable technology more likely, it is a non-optimum process which can prove extremely expensive in terms of time and resources. The quality of the original prototype is always dictated by the accuracy or validity of the designer's conceptualization of the intended users and their tasks and if this is weak, testing may serve less to inform than to reject, and costly resources may be wasted on unnecessary iterations. Even if this is unlikely to occur, and we can, as Landauer (1995) argues, be confident that testing will provide all the information we need, a good first approximation will always save time and money that is in short supply.
In the domain of information usage and digital libraries, such first conceptualizations of usability are often very weak, yet the decisions taken at this point are likely to have major ramifications for the product. Thus, in an attempt to aid designers ( a term I use to include evaluators, task analysts, and all others involved in the messy world of 'design') I have developed the TIME framework.
In developing TIME, an explicit representation was sought of the issues that emerged repeatedly in design contexts involving a variety of hypertext and other electronic documents (e.g., software manuals, process handbooks, research project archives, academic journals). The intention of the framework is to provide those developing digital information resources with a simple way to conceptualize the human factors influencing the usability of the created artifact. TIME is based on seven years of investigations of human information usage from an HCI perspective. This framework was termed TIMS in its original explications (Dillon, 1994) but the revised version reflects a slightly modified labeling of the same components.
The TIME framework is predicated on the following assumptions of human information usage:
(i) Humans explore and use information in a goal-directed manner to 'satisfice' the demands of their tasks
(ii) Humans form models of the structure of and relationship between information units. Repeated formation and application of such models leads to the formation of schematic forms and ultimately, document genres.
(iii) Human information usage consists in part of physical manipulation of information sources
(iv) Human reading at the level of word and sentence perception is bounded in part by the established laws of cognitive psychology
(v) Human information usage occurs in contexts that enable the user to apply multiple sources of knowledge to the information task being performed.
In form, TIME is a qualitative framework and is proposed for use as an advanced organizer for design, as a guide for heuristic and expert usability evaluation, and as a means of generating scientific conjectures about the usability of any electronic text. The framework raises for consideration four key factors affecting usability. They are:
1. A task (T) that reflects the reader's needs and uses for the material;
2. An information model (I) that consists of the user's mental model of the information space;
3 The manipulation skills and facilities (M) that support physical use of the material;
4. The ergonomic variables (E) influencing the perceptual processing of words and images.
Human information usage is thus conceptualized as a process involving the abstraction of meaning in accordance with the shifting focus of the user, and embedded in an informational context that impacts all four levels of analysis. The frmaework is applied by walking though a task the user will perform, and applying in turn the TIME activities as they occur. To make this clearer, it is important to explicate the TIME components in more detail.
Users interact with information purposively, to obtain data, to be entertained, to learn etc. To do this they must decide what it is they want to get out of the resources to hand, determine how they will tackle the information space (e.g., browse or read start to finish, follow a link or ignore it for now etc. ). Furthermore, during the task they must review their progress and, if necessary, revise aspects of the task.
This notion of intentionality in usage gives rise to the idea of planning which evidence suggests is relatively gross, taking the form of such intentions as "locate a reference that is relevant", or "see what this site is about". (Dillon, 1994). However, plans can be even vaguer. Reading an academic article to comprehend the full contents seems to be unspecifiable, the reader is more likely to formulate a plan such as 'read it from the start to the finish, skip any irrelevant or trivial bits, and if it gets too difficult jump on or leave it'. Furthermore, such a plan may be modified as the reading task develops e.g., the reader may decide that she needs to re-read a section several times, or may decide that she can comprehend it only by not reading it all. In this sense planning becomes more situated (see e.g., Suchman, 1988) where the reader's plans are shaped by the context of the on-going action and are not fully specifiable in advance.
Information Modeling (I)
Readers possess (from experience), acquire (while using) and utilize a representation of the document's structure that may be termed a mental model of the text or information space. Such models allow readers to identify likely locations for information within the document, to predict the typical contents of a document, to know the level of detail likely to be found and to appreciate the similarities between documents etc. Indeed several years of experimental work suggest this must be the case (see also van Dijk and Kintsch (1983).
The notion of information model has gained currency in discussion of digital documents largely through the use of the navigation metaphor to describe the process users must go through to gain a familiarity with the contents. The information model is based on the user's attempts to organize the information space's contents into a meaningful structure. Where this model is weak, navigational difficulties will occur as the user cannot call on an accurate represnetation of the organization of the information space.
The model is not purely the result of bottom-up processing. Users may have a pre-existing model of how the information is organized and thus what is initially a model becomes, with use, a map of a specific text. Where no model exists in advance, it is hypothesized that a map can be formed directly (though it may require more effort on the part of the user). Similarly, where repeated map formation suggests regularities in the model, then where this is communicated across time and other users, a genre may be formed. In DL research, there are many questions as yes unsanswered about information models.
Interacting with information involves a substantial amount of physical engagement. As I have noted before (Dillon, 1994) with paper, such skills are acquired early in life and are largely transferable from one text form to another. Most readers can use their fingers to keep pages of interest available while searching elsewhere in the document or flicking through pages of text at just the right speed to scan for a particular section, but beyond these actions, manipulation of documents becomes difficult. When one then considers manipulation of multiple documents these limitations are exacerbated.
Large digital documents are awkward to manipulate by means of scrolling or paging alone but 'point and click' facilities and graphical user interface qualities have improved this, at least in terms of speed, although the present author still longs for the digital equivalent of the finger that is less permanent than a bookmark and serves the temporary holding of a place in space. The lack of standards in current electronic information systems means that acquiring the skills to manipulate documents on one system will not necessarily be of any use for manipulating texts on another. Obviously digital systems afford sophisticated manipulations such as searching which can prove particularly useful for certain tasks and render otherwise daunting tasks (such as locating thematically-linked quotations from the complete works of Shakespeare) now manageable in minutes rather than days. Yet electronic search facilities are far from a guarantee of accurate performance.
Ultimately, the goal is to design transparent manipulation facilities that free the users' processing capacity for task completion. Slow or awkward manipulations are certain to prove disruptive to the usage process. The typical delay experienced by web browsers at busy times is the most obvious example of this. The framework raises these issues as essential parts of the information usage process and therefore important ones for designers to consider in the development of electronic text.
Visual Ergonomics (E)
The final element of the framework reflects the visual ergonomics factors. It is at this level of analysis that we deal with the activities most typically described as 'reading' in the psychological literature (e.g., Just and Carpenter 1980). Thus eye movements, fixations, letter/word recognition and other perceptual, linguistic and (low-level) perceptual and cognitive functions involved in extracting meaning from the textual image are properly located at this level.
In terms of design for digital libraries, this level cannot be overlooked. Ability to effectively read electronic text for example is contingent upon image quality (Gould et al, 1987) and any application that fails to address this basic level will not be redeemable to users in terms of content or task support. Thus, the issues at this level can be seen as necessary (though insufficient) determinants of digital library usability.
So far, the basic components of the framework have been described. A schematic representation of the framework is presented in figure 1. As shown, the elements are all related and collectively framed within the social context in which the activity occurs.
Fig. 1 The TIME framework
When users interact with a document they engage in multiple rapid acts involving the various elements in the TIME framework. For example, in a digital newspaper scenario, the user might start by targetting certain information (e.g., sports reports). To gain the fastest answer it is likely that the user will apply her information model if the newspaper structure. This would involve T -> I and I->T exchanges as well as numerable I->M, and M->E->M activities as the user scans the text headings and moves through the newspaper. Finally, if succesful, the user will locate the relevant section and proceed to read the sports scores, at which point the major human factors issue is one of screen ergonomics as the user reads the text.
As this happens, it is possible for the designer to analyse the task of the user in such a scenario in order to get a sense of the various proportions of the user's efforts that are placed at each level. Thus a task that requires multiple searches and quick location of details will place manifest different sequences of T->I->M->E interaction than a task involving once off location followed by lengthy reading.
To see this in action, consider the following example. In this case, the evaluator wishes to apply TIME to the evaluation of a digital newspaper. One scenario she is testing is the case of a user looking for a sports score. To use TIME, the idea is to first specify the activities that the user must perform to complete the task. Then, for each of these activities, the evaluator needs to consider the various I, M, and E issues associated with each activity.
To proceed, the evaluator must first list the steps a user must take to complete a task. In this case there are 4 basic steps that a user must follow to complete this task. For each step, the evalautor then examines the Information Model (I), Manipulation (M) and Ergonomic (E) issues that are likely to impact the user. This information forms the evaluators best guess as to the likely processes followed by the user in dealing with the information space in multi-levelled terms. The information can be simply tabulated and augmented with comments as to user requirements and usability issues noted by the evaluator, to form a TIME frame, as in Table 1.
Table 1 - a TIME frame for the expert evaluation of a prototype digital newspaper.
Why a multi-levelled framework?
TIME is multilevelled in that it brings together concerns with human response at all levels - from the physical to the socio-cultural. To use a DL environment requires engagement at the physical and perceptual levels to manipulate and view information. Cognitive processes are involved in comprehending and navigating the information space. Social factors influence the formation of information models and the and the contexts in which DLs are employed. To model information requires users not only to acquire the form of representation generally evaluated as navigation knowledge, they must also develop an appreciation of the content of the information space in semantic terms. To do this, users seems to learn to respond to conventions in information space that they notice with repeated use, and which experienced members of a discourse community utilize to convey their ideas. The term shape seems to more adequately convey this set of representational cues, and as such, invokes social determinants of cognitive representation that cannot be conveniently labeled as cognitive or social, but are best understood as socio-cognitive aspects of interface design (see Dillon (in press) and Dillon and Vaughan, (1997) for a detailed explication of the shape construct).
Similalrly, the task component of the model is not intended to represent user task in a limited idealised form akin to the classic engineering approaches of Card et al (1983) or Drury (1987) but to allow for description of the range of intentions users are likely to bring to an digital library application as they seek to satisfice information needs. In this way it is advocating scenario identification and an awareness of the necessary updating and shifting intentions of the user embedded in a context of information usage.
TIME as design tool
The TIME framework is intended to used as means of understanding interaction in context. It seeks to avoid prescriptive guidelines for design and instead offer designers and implementers a systematic framework within which to think about the interfaces they are providing to their resources.
In the first instance the framework is useful as a guiding principle or type of advance organizer of information that gives the designer an orientation towards design enabling the application of relevant knowledge to bear on the problem. Thus when faced with a design problem requiring digital documents for example, the designer could conceptualize the intended users in TIME terms and thereby guide or inform their prototyping activities.
Secondly, by parsing the issues into elements, the framework facilitates identification of the important ones to address. This framework suggests four major issues to consider in any information context: the user's task and their perception of it; the information model they possess, must acquire or is provided by the artifact; the manipulation facilities they require or are provided; and the actual 'eye-on-document' aspects involved. All are ultimately important, though depending on the task and the size of the information space, some may be more important than others (e.g., for very short texts of one screen or less, manipulation facilities and information models might be less important than visual ergonomics, although task contingencies must be considered here before such a conclusion could be reached).
In the third instance the framework provides a means for ensuring that all issues relevant to the design of electronic text are considered. It is not enough that research or analysis is carried out on text navigation and developers ignore image quality or input devices (and vice-versa). A good electronic digitla library interface will address all issues.
The above applications consider the uses at the first stages of system development. However the framework also has relevance to later stages of the design process such as evaluation. In such a situation the framework user could assess a system in terms of the four elements and identify potential weaknesses in a design. This would be a typical use for expert evaluation, perhaps the most common evaluation technique in HCI. This form of use has been most commonly made of the framework by the present author.
Certainly an immediate application is to enhance our critical awareness of the claims and findings in the literature on HCI. For example, it is not untypical to hear statements to the effect that 'hypertext is better than paper' or 'reading from paper is faster than reading from screens' TIME suggests caution in interpreting such statements. If human information usage involves all elements in the framework then the only worthwhile statements are those that include these. Thus, the statement 'hypertext is better than paper' is, according to the TIME perspective, virtually useless since it fails to mention crucial aspects such as the context of use, tasks for which it is better, the nature of the information models required by readers to make it better or worse, the manipulation facilities involved, or the image quality of the screen that effects the standard reading processes of the reader. We know all these variables are crucial since we have over 15 years of research investigating them and trying to untangle their varying effects.
Thus, TIME suggests statements about interwction with digital documents, to be of value, need to be complete and make explicit each of the elements in its claim. Thus, a more useful statement would be: "For reading lengthy texts for comprehension, for which readers have a well developed information model, on a scrolling window, mouse-based system, screens of more than 40 lines are better than those of 20 lines, though both are worse than good quality paper"
Notice that the truth content of the statement is not what is at issue here. It is the completion of the statement that is important. Incomplete statements (ones not making reference to all elements of the TIME framework) are too vague to test since there are unlimited sources of variance that could exist under the general headings "hypertext", "paper" or "is better than". A complete statement might be wrong, but at least it should be immediately testable or refutable given the existing body of empirical data the field has amassed.
Coupling these activities to the operartionalisation of usability is another possible application of the framework. It is difficult to meaningfully derive criteria for effectivenss, efficiency and satisfaction for users of an application without some informed view of the tasks they wil be performing and the experience and skills thay possess. Stepping through the elements of the TIME framework, particualrly the task and information modle components focusses attention on key issues that affect these criteria setting operations.
The usability of the interfaces we provide to digital library applications is crucial. While the field of HCI has provided many recommendations on design, it is difficult to tease out relevant ones to any given design project and any guideines that are employed need to be contextually interpreted for relevance. The TIME framework is aimed at providing interface developers and digital library implementers with a means of parsing the issues into tractable points.
Coupled with an operational approach to defining usability, this framework can move digital library application design beyond the purely empirical approach that works, if employed correctly, but is time-consuming and costly. By affording insight into the issues that drive interaction, TIME can be used to derive relevant guidelines from the literature, support the definition of appropriate usability criteria, and support expert evaluation of the interface. All applications serve to speed up the design-test loop that is necessary for acceptable technology interfaces in the digital library domain to be built.
Bailey, R. (1995) Performance vs. Preference. Proc. of the Hum Fac Soc Annual Conference. New York: HFES, 316-320.
Card, S., Moran, T. and Newell, A. (1983) The psychology of human-computer interaction. Hillsdale NJ: Lawrence Erlbaum Associates.
Carroll, J. and Campbell, R. (1989) Artifacts as psychological theories: the case of HCI. Behaviour and Information Technology, 8(4) 247-256.
Chapanis, A. (1988) Some generalizations about generalizations. Human Factors, 30(3) 253-268.
Chui, M. and Dillon, A. (1997) Who's zooming whom? Attunement to animation at the interface. Journal of the American Society for Information Science, 48 (11) 1067-1072.
van Dijk and Kintsch (1983). Strategies for Dicsource Comprehension. New York: Academic Press.
Dillon, A. (1994) Designing Usable Electronic Text: Ergonomics Aspects of Human Information Usage. London: Taylor and Francis
Dillon, A. (1996) Myths, Misconceptions And An Alternative Perspective On Information Usage And The Electronic Medium. In: Rouet, J-F., Levonen, J.J., Dillon, A.P. & Spiro, R.J. (Eds.) Hypertext and Cognition. Hillsdale, NJ: Lawrence Erlbaum Associates
Dillon, A (in press) Spatial semantics. Journal of the American Society for Information Science, 47,10, 786-788.
Dillon, A. and Vaughan, M. (1997) It's the journey and the destination: Shape and the emergent property of genre in digital documents. New Review of Multimedia nad Hypermedia Vol 3, (1997) 91-106.
Drury, C. (1987) Task Analysis. In: G. Salvendy (ed.) Handbook of Human Factors Hillsdale, NJ: Lawrence Erlbaum Associates
Eason, K. (1988) Information Technology and Organisational Change. London: Taylor and Francis.
Gould, J.D., Alfaro, L., Barnes, V., Finn, R., Grischkowsky, N. and Minuto, A. (1987) Reading is slower from CRT displays than from paper: attempts to isolate a single variable explanation. Human Factors, 29(3) 269-299.
Gray, W. et al (1992) The precis of project Ernestine or An overview of a validation of GOMS. Proceedings of the ACM SCIGCHI Annual Conference CHI'92, New York: ACM 307-312.
Just, M.A. and Carpenter, P. (1980) A theory of reading: from eye movements to comprehension. Psychological Review, 87(4), 329-354.
Landauer, T. (1995) The Trouble with Computers. Boston: MIT Press.
Landauer, T. Egan, D, Remde, J., Lesk, M., Lochbaum, C. and Ketchum (1993) Enhancing the usability of text through computer delivery and formative evaluation: The SuperBook Project. In C. McKnight, A. Dillon and J. Richardson (eds.) Hypertext: A Psychological Perspective, London: Ellis Horwood, 71-136.
McKnight, C., Dillon, A. and Richardson, J. (1991) Hypertext in Context. Cambridge: Cambridge University Press
Nielsen, J. (1993) Usability Engineering. Boston: AP Professional.
Norman, K. (1991) The Psychology Of Menu Selection. Norwood: Ablex.
Rothkopf, E. Z. (1971) Incidental memory for location of information in text. Journal of Verbal Learning and Verbal Behavior, 10, 608-613.
Shackel, B. (1991) Usability - Context, Framework, Definition, Design and Evaluation. In B. Shackel and S. Richardson (eds.) Human Factors for Informatics Usability. Cambridge: Cambridge University Press. 21-37.
Suchman, L. (1987) Plans and Situated Action.Cambridge: Cambridge University Press.