How do humans and computers interact?

1.3.1 The Traditional Concept of Usability

Human–computer interaction is what happens when a human user and a computer system, in the broadest sense, get together to accomplish something. Usability is that aspect of HCI devoted to ensuring that human–computer interaction is, among other things, effective, efficient, and satisfying for the user. So usability1 includes characteristics such as ease of use, productivity, efficiency, effectiveness, learnability, retainability, and user satisfaction (ISO 9241-11, 1997).

1.3 SCIENTIFIC FRAGMENTATION

HCI has been a successful technological and scientific undertaking. It achieved an effective integration of software engineering and the human factors of computing systems through the concepts and methods of cognitive science. In doing so, it helped to broaden and develop cognitive science itself. No one could have anticipated in 1980 just how HCI would develop. And we cannot know its future course now. However, the progress of the past two decades highlights specific current challenges.

An ironic downside of the inclusive multidisciplinarity of HCI is fragmentation. This is in part due merely to the expansion of the field and its scientific foundations. In the 1980s, it was reasonable to expect HCI professionals, particularly researchers, to have a fairly comprehensive understanding of the concepts and methods in use. Today, it is far more challenging for individuals to attain that breadth of working knowledge. There are too many theories, too many methods, too many application domains, too many systems. Indeed, the problem of fragmentation may be a bit worse than it has to be. Some HCI researchers, faced with the huge intellectual scope of concepts and approaches, deliberately insulate themselves from some portion of the field's activity and knowledge. This tension between depth and breadth in scientific expertise is not unique to HCI, but it clearly undermines the opportunity for multidisciplinary progress.

Fragmentation is also manifest among HCI practitioners. In the early 1980s, in a smaller and narrower HCI community, there was a close coordination of research and practice. The relationship between research and practice in this early period was quite orthodox—the view being that researchers developed concepts, methods, technologies, and prototypes, and practitioners applied and developed them in products. This is actually not a very good model of effective technology development, and it has been supplanted by a more interactive view in which practice plays a more central role in articulating requirements for theory and technology and in evaluating their efficacy in application. However, playing more creative roles in multidisciplinary technology development demands much more of practitioners. It demands that they understand the intellectual foundations of HCI, not merely how to manipulate the tools and methods constructed on those foundations. Although there are many encouraging examples of multidisciplinary HCI practice leading research in highly creative ways, practitioners also often manage the intellectual scope of concepts and approaches by deliberately isolating themselves from some portion of the field's foundations. Ironically, because HCI practice has diversified so rapidly and has incorporated so many new professionals, average expertise among practitioners has never been lower.

The intrinsic difficulties of making sense of a vast and diverse science base are exacerbated by engineering exigencies in contemporary software and systems. During the 1990s, development cycles for new products and services were dramatically compressed. Thus, even among researchers and practitioners with the inclination and skills to pursue a multidisciplinary program of HCI research and application, external factors such as schedule, budget, and compatibility with standard solutions often prevail. This has created an often-overwhelming pressure to streamline methods and techniques so that they can be conveyed to novices in a half-day tutorial, and then put into practice by tutees the very next day. Of course, it is important to manage cost-benefit tradeoffs in methods and techniques, to ensure that the benefits expected merit the effort required. But the pressure to simplify has done violence to some theory-based approaches: Cognitive modeling is frequently conflated with keystroke-level counting, ethnography is conflated with any observational study, and thinking-aloud protocols are conflated with concurrent participant interviews. The pressure to simplify has also led to a higher-than-warranted reliance on checklists and guideline-based usability methods, many of which are grounded in common sense and face validity but not in specific research.

Like most of computing, HCI is a relatively small research community amid a huge community of practitioners seeking to apply HCI knowledge and techniques to practical problems within schedule and budget constraints. This demographic is itself a reflection of the field's success through the past two decades, but it creates pressures to streamline and disseminate concepts and techniques. On the one hand, there is enormous demand for researchers to apply and disseminate concepts and techniques. And on the other hand, there is little enthusiasm or support for researchers to articulate the relationships among concepts and techniques. An example is the current prevalence—both explicit and implicit—of “quick and dirty” ethnography, but without a comprehensive analysis of the consequences of various ethnographic frameworks.

To the extent that the success of HCI through the past two decades was due to its ambitiously multidisciplinary application, fragmentation is a threat to future development. Put more positively, HCI will fully use the leverage of its multidisciplinary science foundation only if it can continually synthesize a coherent methodological framework from that foundation. In the “golden age” of HCI as cognitive science, the underlying rubric for this coherence was the representational theory of mind. This foundation was highly successful in underwriting an engineering practice. But it also helped to raise further questions and to broaden HCI through further disciplinary perspectives. As the multidisciplinary program for HCI science continues to develop, synthesizing a comprehensive and coherent methodological framework will become more challenging but also potentially more powerful.

1.1 Introduction

Although HCI emerged in the 1980s, it owes a lot to older disciplines. The most central of these is the field of human factors, or ergonomics. Indeed, the name of the preeminent annual conference in HCI—the Association for Computing Machinery Conference on Human Factors in Computing Systems (ACM SIGCHI)—uses that term. SIGCHI is the special interest group on computer-human interaction sponsored by the ACM.2

Human factors is both a science and a field of engineering. It is concerned with human capabilities, limitations, and performance, and with the design of systems that are efficient, safe, comfortable, and even enjoyable for the humans who use them. It is also an art in the sense of respecting and promoting creative ways for practitioners to apply their skills in designing systems. One need only change systems in that statement to computer systems to make the leap from human factors to HCI. HCI, then, is human factors, but narrowly focused on human interaction with computing technology of some sort.

That said, HCI itself does not feel “narrowly focused.” On the contrary, HCI is tremendously broad in scope. It draws upon interests and expertise in disciplines such as psychology (particularly cognitive psychology and experimental psychology), sociology, anthropology, cognitive science, computer science, and linguistics.

Figure 1.1 presents a timeline of a few notable events leading to the birth and emergence of HCI as a field of study, beginning in the 1940s.

4.5 Visual Representation and Interaction

For visualizing time-evolving graphs, it is important to choose a good visual representation to show them in a presentable and understandable format. These visual representations should reduce visual clutter and minimize temporal aliases for node positions across time and maximize readability and scalability. Selecting a visual representation for time-evolving graphs is restricted by the data at hand, size of the graph, amount of data to visualize, purpose of the visualization, etc. Some of the visual representations are limited by graph layouts as it is difficult to find automatic layouts for static graphs, and to do that for every time stamp is an enormous task. Other extensions to the animation-based and timeline-based visualization techniques include 3D visualization (Gaertler and Wagner, 2005; Kumar and Garland, 2006), hybrid representations combining animation, and timeline drawings. Several application-specific visualizations are available in the literature, including time line trees (Burch et al., 2008), tree maps (Hao et al., 2005), icicle plots (Tekušová and Schreck, 2008), node–link diagrams with time series (Saraiya et al., 2005), and time arc trees (Greilich et al., 2009).

HCI enables users to interactively browse the data set to discover hidden insights. An effective HCI is equally important as visual representation for a good visual analytics framework. These HCIs should enable the user to have control over what and how they want to see and to define the flow and parameters of decision informatics. Recent studies (Heer and Shneiderman, 2012; Kerren and Schreiber, 2012; Yi et al., 2007) provide taxonomies for visual interaction techniques to help better understand and improve visual analytics designs. Interactions with the visual representations are divided into three high-level categories:

Data and view specifications: An HCI should allow the user to reconfigure views based on attributes of interest, to filter portions of the graph, and to derive simple analytics using statistical computations.

View manipulations: A user should be able to select, highlight, and bookmark portions of the graph either by manual selection or through search criterion, and to navigate and explore over graphs using zooming, magic and fish eyed lenses, panning, etc. The HCI should allow the user to coordinate and organize multiple views for easy comparisons of results from different interactions.

Process and provenance: Visual analytics systems should record different interactions for fast recall or revisiting of past analyses. They should also support multiuser collaboration, reporting, and sharing of views, interactions, and results.

The other important aspect of visualization is rendering the graph to display large-scale data sets. GPU-based rendering is becoming increasingly common. Gephi provides a time-sliding-based tool to navigate a time-varying graph. There are several other rendering techniques for multivariate graphs that can be applied for time-varying graphs, graph visualization libraries and tools, and network visualization tools available for use. The choice of tools depends on the size, scale, the nature of the graphs, the type of analysis (flow-based, relationships, clusters, cliques), and the platform for visualization (desktop or web browser). Some of the widely used visualization desktop visualization tools are Gephi (Gephi, 2016), Cytoscape (Cytoscape, 2016), Palantir (Palantir, 2016), ComVis (Matković et al., 2008), and Dato (GraphLab) (Dato (GraphLab), 2016). There are also several web-based visualization libraries that include D3.js (D3.js, 2016), Sigma.js (Sigma.js, 2016), and Vivagraph.js (VivaGraph.js, 2016). More detailed lists of visualization tools are available in kdnuggets (2016) and blog (2016).

1.5 Understanding HCI Research Methods and Measurement

HCI research requires both rigorous methods and relevance. It is often tempting to lean more heavily towards one or the other. Some other fields of research do focus more on theoretical results than on relevance. However, HCI research must be practical and relevant to people, organizations, or design. The research needs to be able to influence interface design, development processes, user training, public policy, or something else. Partially due to the philosophies of the founders of the field, HCI has had a historic focus on practical results that improve the quality of life (Hochheiser and Lazar, 2007). Is there a tension sometimes between researchers and practitioners? Absolutely. But all HCI research should at least consider the needs of both audiences. At the same time, the research methods used (regardless of the source discipline) must be rigorous and appropriate. It is not sufficient to develop a new computer interface without researching the need for the interface and without following up with user evaluations of that interface. HCI researchers are often placed in a position of evangelism where they must go out and convince others of the need for a focus on human users in computing. The only way to back up statements on the importance of users and human-centered design is with solid, rigorous research.

Due to this interdisciplinary focus and the historical development of the field, there are many different approaches to measurement and research currently used in the field of HCI. A group of researchers, all working on HCI-related topics, often disagree on what “real HCI research” means. There are major differences in how various leaders in the field perceive the existence of HCI. Be aware that, as an HCI researcher, you may run into people who don't like your research methods, are not comfortable with them, or simply come from a different research background and are unfamiliar with them. And that's OK. Think of it as another opportunity to be an HCI evangelist. (Note: As far as we know, the term “interface evangelist” was first used to describe Bruce Tognazzini. But we really think that the term applies to all of us who do HCI-related work.) Since the goal of this book is to provide a guide that introduces the reader to the set of generally accepted empirical research practices within the field of HCI, a central question is, therefore, how do we carry out measurement in the field of HCI research? What do we measure?

In the early days of HCI research, measurement was based on standards for human performance from human factors and psychology. How fast could someone complete a task? How many tasks were completed successfully, and how many errors were made? These are still the basic foundations for measuring interface usability and are still relevant today. These metrics are very much based on a task-centered model, where specific tasks can be separated out, quantified, and measured. These metrics include task correctness, time performance, error rate, time to learn, retention over time, and user satisfaction (see Chapters 5 and 10 for more information on measuring user satisfaction with surveys). These types of metrics are adopted by industry and standards-related organizations, such as the National Institute of Standards and Technology (in the United States) and the International Organization for Standardization (ISO). While these metrics are still often used and well-accepted, they are appropriate only in situations where the usage of computers can be broken down into specific tasks which themselves can be measured in a quantitative and discrete way.

Shneiderman has described the difference between micro-HCI and macro-HCI. The text in the previous paragraph, improving a user's experience using well-established metrics and techniques to improve task and time performance, could be considered micro-HCI (Shneiderman, 2011). However, many of the phenomena that interest researchers at a broader level, such as motivation, collaboration, social participation, trust, and empathy, perhaps having societal-level impacts, are not easy to measure using existing metrics or methods. Many of these phenomena cannot be measured in a laboratory setting using the human factors psychology model (Obrenovic, 2014; Shneiderman, 2008). And the classic metrics for performance may not be as appropriate when the usage of a new technology is discretionary and about enjoyment, rather than task performance in a controlled work setting (Grudin, 2006a). After all, how do you measure enjoyment or emotional gain? How do you measure why individuals use computers when they don't have to? Job satisfaction? Feeling of community? Mission in life? Multimethod approaches, possibly involving case studies, observations, interviews, data logging, and other longitudinal techniques, may be most appropriate for understanding what makes these new socio-technical systems successful. As an example, the research area of Computer-Supported Cooperative Work (CSCW) highlights the sociological perspectives of computer usage more than the psychological perspectives, with a focus more on observation in the field, rather than controlled lab studies (Bannon, 2011).

The old methods of research and measurement are comfortable: hypothesis testing, statistical tests, control groups, and so on. They come from a proud history of scientific research, and they are easily understood across many different academic, scientific, and research communities. However, they alone are not sufficient approaches to measure all of today's phenomena. The same applies to the “old standard” measures of task correctness and time performance. Those metrics may measure “how often?” or “how long?” but not “why?” However, they are still well-understood and well-accepted metrics, and they allow HCI researchers to communicate their results to other research communities where the cutting-edge tools and research methods may not be well-understood or well-accepted.

You may not be able to use experimental laboratory research to learn why people don't use technology. If you want to examine how people use portable or mobile technology such as smart phones and wearable computing, there are limitations to studying that in a controlled laboratory setting. If you want to study how people communicate with trusted partners, choose to perform business transactions with someone they don't know on another continent (as often happens with Ebay), or choose to collaborate, you need to find new ways of research and new forms of measurement. These are not research questions that can be answered with quantitative measurements in a short-term laboratory setting.

Consider Wikipedia, a collaborative, open-source encyclopedia. Currently, more than five million articles exist in English on Wikipedia, with an estimate of 70,000 active contributors (https://www.wikipedia.org), who spend their own time creating and editing Wikipedia entries. What causes them to do so? What do they get out of the experience? Clearly, task and time performance would not be appropriate metrics to use. But what metrics should be used? Joy? Emotion? A feeling of community? Lower blood pressure? This may not be a phenomenon that can be studied in a controlled laboratory setting (Menking and Erickson, 2015). The field of HCI has begun to apply more research methods from the social sciences, and we encourage the reader to start using some new research approaches that are not even in this textbook! Please be aware that people from other disciplines, as well as your “home discipline,” will probably challenge the appropriateness of those research methods!

3.6 Human-Computer interaction

Human-Computer Interaction (HCI) is a subfield within computer science concerned with the study of the interaction between people (users) and computers and the design, evaluation and implementation of user interfaces for computer systems that are receptive to the user's needs and habits. It is a multidisciplinary field, which incorporates computer science, behavioral sciences, and design. A central objective of HCI is to make computer systems more user-friendly and more usable. Users interact with computer systems through a user interface, which consists of hard- and software that provides means of input, allowing users to manipulate the system, and output, allowing the system to provide information to the user. The design, implementation and evaluation of interfaces is therefore a central focus of HCI.

It is recognized in HCI that good interface design presupposes a good theory or model of human-computer interaction, and that such a theory should be based in large part on a theory of human cognition to model the cognitive processes of users interacting with computer systems [Peschl and Stary, 1998]. Such theories of human cognition are usually derived from cognitive psychology or the multi-disciplinary field of cognitive science. Whereas philosophers have rarely studied human-computer interaction specifically, they have contributed significantly to theorizing about cognition, including the relation between cognition and the external environment, and this is where philosophy relates to HCI.

Research in HCI has initially relied extensively on classical conceptions of cognition as developed in cognitive psychology and cognitive science. Classical conceptions, alternatively called cognitivism or the information-processing approach, hold that cognition is an internal mental process that can be analyzed largely independently of the body of the environment, and which involves the manipulation of discrete, internal states (representations or symbols) that are manipulated according to rules or algorithms [Haugeland, 1978]. These internal representations are intended to correspond to structures in the external world, which is conceived of as an objective reality fully independent of the mind. Cognitivism has been influenced by the rationalist tradition in philosophy, from Descartes to Jerry Fodor, which construes the mind as an entity separate from both the body and the world, and cognition as an abstract rational, process. Critics have assailed cognitivism for these assumptions, and have argued that cognitivism cannot explain cognition as it actually takes place in real-life settings. In its place, they have developed embodied and situated approaches to cognition that conceive of cognition as a process that cannot be understood without intimate reference to the human body and to the interactions of humans with their physical and social environment [Anderson, 2003]. Many approaches in HCI now embrace an embodied and/or situated perspective on cognition.

Embodied and situated approaches share many assumptions, and often no distinction is made between them. Embodied cognition approaches hold that cognition is a process that cannot be understood without reference to the perceptual and motor capacities of the body and the body's internal milieu, and that many cognitive processes arise out of real-time goal-directed interactions of our bodies with the environment. Situated cognition approaches hold that cognitive processes are co-determined by the local situations in which agents find themselves. Knowledge is constructed out of direct interaction with the environment rather than derived from prior rules and representations in the mind. Cognition and knowledge are therefore radically context-dependent and can only be understood by considering the environment in which cognition takes place and the agent's interactions with this environment.

Embodied and situated approaches have been strongly influenced by phenomenology, especially Heidegger, Merleau-Ponty and the contemporary work of Hubert Dreyfus (e.g., [Winograd and Flores, 1987; Dourish, 2001; Suchman, 1987]). Philosophers Andy Clark and David Chalmers have developed an influential embodied/situated theory of cognition, active externalism, according to which cognition is not a property of individual agents but of agent-environment pairings. They argue that external objects play a significant role in aiding cognitive processes, and that therefore cognitive processes extend to both mind and environment. This implies, they argue, that mind and environment together constitute a cognitive system, and the mind can be conceived of as extending beyond the skull [Clark and Chalmers, 1998; Clark, 1997]. Clark uses the terms “wideware” and “cognitive technology” to denote structures in the environment that are used to extend cognitive processes, and he argues that because we have always extended our minds using cognitive technologies, we have always been cyborgs [Clark, 2003]. Active externalism has been inspired by, and inspires, distributed cognition approaches to cognition [Hutchins, 1995], according to which cognitive processes may be distributed over agents and external environmental structures, as well as over the members of social groups. Distributed cognition approaches have been applied to HCI [Hollan, Hutchins and Kirsh, 2000], and have been especially influential in the area of Computer Supported Cooperative Work (CSCW).

Brey [2005] has invoked cognitive externalist and distributed cognition approaches to analyze how computer systems extend human cognition in humancomputer interaction. He claims that humans have always used dedicated artifacts to support cognition, artifacts like calendars and calculators, which HCI researcher Donald Norman [1993] has called cognitive artifacts. Computer systems are extremely versatile and powerful cognitive artifacts that can support almost any cognitive task. They are capable of engaging in a unique symbiotic relationship with humans to create hybrid cognitive systems in which a human and an artificial processor process information in tandem. However, Brey argues, not all uses of computer systems are cognitive. With the emergence of graphical user interfaces, multimedia and virtual environments, the computer is now often used to simulate environments to support communication, play, creative expression, and social interaction. Brey argues that while such activities may involve distributed cognition, they are not primarily cognitive themselves. Interface design has to take into account whether the primary aim of applications is cognitive or simulational, and different design criteria exist for both.

Human–computer interaction (HCI) is the study of how people use technological artifacts, and their design. Unified cognitive architectures such as GOMS and Soar, derived from artificial intelligence, have proven useful theoretically, but too detailed for general application in design. The recognition that design flaws had to be identified early in the design process has lead to the idea of user-centered design, with techniques such as QOC supporting the design process, and the development of evaluation methods, such as cognitive walkthroughs and codifications of practical advice in the form of guidelines. To help designers understand why usability problems might have arisen, cognitive psychology theories are being couched as supportive evaluation methods. One such approach, ICS, is an example of a trend away from the dyadic, turn-taking relationship between a single user and a single computer, towards a more complex view of computers as part of our everyday environment. This trend is increasing the involvement of social and organizational theorists. From another perspective, computer scientists see systems as built up from ‘interactors’ and argue that the user's cognitive processes can be modeled as just another set of interactors. Syndetic models examine the overall system of human and computer interactors. In general, HCI is focusing less on the individual, has a great deal to gain from other social sciences, and has the potential to provide insight into human behavior in return.

Abstract

HCI researchers rely heavily on software tools for data collection. Widely available general-purpose software tools such as web servers and proxies can track page requests, providing detailed logs of user activity, complete with timestamps. Applications ranging from GUI desktops to email programs, web browsers, and web tools leave detailed records of user interactions. Activity-logging tools can track keystrokes and mouse movements, both within web pages and more generally. Instrumented software tools add activity tracking facilities to existing software, while custom research tools support presentation of tasks and collection of data tuned to the needs of specific studies. Selection of appropriate techniques requires careful selection of data collection techniques appropriate for study goals and granularity of required data.

Making Meaning

HCI scholar/scientist Paul Dourish has hypothesized a different take on Interaction Design, that he terms as “embodied interaction,” a new model of interpreting interaction that extends recent HCI research trends in “tangible” and “social” computing.84 Dourish's argument is based upon the philosophical framework of phenomenology which is the study of experience and existence, that are intuitively felt and known by factual presence in the world. Dourish contends that embodiment is more than a physical property but is about social presence and participatory status in the world, having an (inter) active role in changing and becoming. Everyday engagement in daily activities and task completion is another core tenet; the setting of action defines the value and manner of the action. Thus meaning emerges from the participation of an individual agent with some object within a setting—a constant negotiation or conversation unfolding. It is formed continuously and interactively, in real-time action/location; meaning is not simply projected or found but instead created and shared through engagement with the artificial.85 This is a profound view of interaction that shifts the emphasis from the designer crafting the argument, or the interpretation of images, towards the place of action between the user and the object in question, given a situation and the particular lifestyle of the user.

This view encourages the designer to regard design as a participatory activity, not simply dictating to the user, but allowing the user to evolve and shape the encounter so it is a co-creative opportunity. Indeed, this view presupposes that the user can manipulate or improvise the design to suit her needs at the moment, as recently suggested by IDEO designer Janet Fulton Suri, in her account of everyday actions, Thoughtless Acts. Suri's work explores what occurs when ordinary objects are re-cast for impromptu purposes—for example, using your suitcase as a seat at an airport internet kiosk.

Another example to consider: Videogame interaction is a highly complex form of communication and engagement, whose meaning arises from the immediate, real-time encounter between the player, the controller, the game console, and the video imagery on the TV display. There is a coalescence of game play, game mechanics, and game interface that constitute the total value of the interaction, its meaning in terms of responsiveness of game interactivity and how it fits within situation/context of leisurely activity. There is learning, pleasure, frustration, and overall struggle and resolution in that continuous, unfolding moment of participation.

Thus, in summary, through the intersection of interaction and language, design becomes a platform for communication. Viable, actionable communication can occur from a variety of viewpoints: rhetorical, semiotic, or phenomenal. There are certainly others but these specific views sufficiently capture key issues of influence, interpretation, and engagement that characterize an interaction. In guiding the designer who seeks an effective communication-oriented solution, these views parcel out finer issues for debate and iteration. These are simply ways to perceive how meaning comes to be in interaction, when regarded as a communicative activity. In actual practice, however, an interactive encounter (and thus meaning itself) combines all three views into a dynamic, self-sufficient, whole user experience.

We have taken a path through the nexus of interaction and language to understand how to create products that deliver positive value to users, and thereby implicitly suggest a broader cultural backdrop of experience. Interaction shapes the perception of reality. A coherent and consistent system of interactions within the framework of design suggests a language of relationship building between people (user + designer, user + other users) mediated by the designed artifice. Value and meaning are deliberated, interpreted, and created via the interactive encounter, at multiple levels: emotional, cognitive, physical, visual. This activity (construed as a conversation or dialogue) characterizes the user experience of an artifact, which can proliferate and aggregate to impact society and culture at large, shaping values, norms, beliefs, attitudes, expectations, or standards of what is acceptable or appropriate. One's way of life or lifestyle itself can be influenced by well-informed Interaction Designs, to yield a satisfying, memorable quality of experience—one that can be shared, repeated, and enhanced.

Copyright for this article is held by Uday Gajendar; reprinted here with permission.